Strip unnecessary security contexts on Windows #93475

ravisantoshgudimetla · 2020-07-27T16:13:38Z

What type of PR is this?

/kind bug
/kind cleanup

What this PR does / why we need it:
As of now, the kubelet is passing the security context to container runtime even
if the security context has invalid options for a particular OS. As a result,
the pod fails to come up on the node. This error is particularly pronounced on
the Windows nodes where kubelet is allowing Linux specific options like SELinux,
RunAsUser etc where as in documentation,
we clearly state they are not supported. This PR ensures that the kubelet strips
the security contexts of the pod, if they don't make sense on the Windows OS.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Strip unnecessary security contexts on Windows

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig windows

ravisantoshgudimetla · 2020-07-27T16:13:50Z

/sig node

sjenning · 2020-07-27T20:44:34Z

pkg/securitycontext/util.go

extra space causing gofmt failure

ddebroy

One overall comment on best place to perform the desired goal of this PR: Strip unnecessary security contexts on Windows.

ddebroy · 2020-07-27T23:18:04Z

pkg/securitycontext/util.go

DetermineEffectiveSecurityContext returns right away if the checks above are satisfied. In that case, if the effectiveSc or containerSc happens to have the members of a security context that do not apply to Windows, they will continue to be returned (and will not be stripped as below).

Since the desired action is specific to Windows, have you considered cleaning up the security contexts in the Windows caller? That would:

Allow the early/optimized exit paths above to continue to work.

Allow DetermineEffectiveSecurityContext to be invoked from a cluster-wide controller context (if needed in the future) rather than the node-specific kubelet context. The runtime.GOOS != "windows" checks assumes DetermineEffectiveSecurityContext will never be invoked from a cluster-wide controller context.

Doing this work in the windows specific code makes sense. We have run into a few other places where this has come up with security context: #92355. In the future there could be a place where some of the these attributes might be available if a form of privileged containers becomes available.

The move to doing the checks in windows specific code makes the resulting logic here much simpler over adding complex if statements which will become hard to maintain and easily introduce bugs.

I initially thought about it, however we have to call DetermineEffectiveSecurityContext first and then resetting the values at the call site which did not make much sense to me. Having said that, I am fine making the change, so that we can have everything within the Windows caller to improve readability.

ddebroy · 2020-07-27T23:50:53Z

pkg/securitycontext/util.go

The effectiveSc != nil && containerSc != nil checks seem to be taken care of by the checks above but I think this is safer code.

pjh · 2020-07-28T16:57:15Z

Can we add a test to verify that the desired behavior works once this PR is merged?

yliaog · 2020-07-28T17:02:35Z

@ravisantoshgudimetla what recent change caused this pod crash behavior?

michmike · 2020-07-28T17:13:29Z

/milestone v1.20

jsturtevant · 2020-07-28T17:54:42Z

pkg/securitycontext/util.go

I don't think this one should be overridden due to the conversation on #92355 (comment) where we decided to check that the ContainerAdministrator is not set for a RunAsNonRoot check.

jsturtevant · 2020-07-28T17:55:26Z

pkg/securitycontext/util.go

same as above via: #92355 (comment)

jsturtevant · 2020-07-28T17:59:13Z

Can we add a test to verify that the desired behavior works once this PR is merged?

Is there a reason this PR should not add tests for this iteration instead of after the fact? At the very least should have unit tests for this behavior. Having an e2e test for this would also be valuable to verify the the behavior does improve.

pjh · 2020-07-28T18:59:53Z

Is there a reason this PR should not add tests for this iteration instead of after the fact?

Yeah that's what I meant, I just phrased it confusingly :)

ravisantoshgudimetla · 2020-07-28T23:35:22Z

@ravisantoshgudimetla what recent change caused this pod crash behavior?

This is not something new. This behavior was there from a long time

ravisantoshgudimetla · 2020-07-28T23:37:03Z

Thank you @ddebroy @jsturtevant @pjh for your reviews. I addressed your comments. I added a test but I am not sure, if it is in right place. PTAL.

ddebroy · 2020-08-04T16:54:36Z

/test pull-kubernetes-integration

wawa0210 · 2020-08-05T14:36:22Z

I am glad to see this pr.I am very excited to see this pr, I have some questions to ask:

Currently pr is just Strip unnecessary security contexts on Windows, and does not apply some of the windows' own te'x
There is currently a bug in windows. If we specify the runasuser of the pod, the windows sandbox is not used, but directly uses the ContainerAdministrator identity to run the container

It seems that the current pr does not really solve these problems

If we want to solve these problems, we need to run different logics according to the operating system when sanbox is created, and at the same time pass the identity information to cri

I am also trying to fix this logic recently. I mentioned an issue #92963 and a pr #93299 before(wip). Can you help discuss whether this method is ok

ravisantoshgudimetla · 2020-08-06T00:31:00Z

@wawa0210 - Thank you for reviewing the PR.

Currently pr is just Strip unnecessary security contexts on Windows, and does not apply some of the windows' own te'x

Please look at the comment - #93475 (comment). It is my understanding that for the Windows pods there is no need to have sandbox level security Constraints. If we want to push the securityContext outside from Container Security Context to sandbox security constraints it needs a broader discussion as it needs api changes and beyond the scope of current PR

There is currently a bug in windows. If we specify the runasuser of the pod, the windows sandbox is not used, but directly uses the ContainerAdministrator identity to run the container

The first question that we need to ask is do we need container sandbox for Windows considering all the security options are at the container Security contexts. What does the new WindowsSandboxConfig offer - just the security options, for example in case of linux we need to identify parent cgroup etc. If it is just the security options there is no need to define new api around it.

@ddebroy has created #93580 to remove the dependency on the LinuxContainerConfig, as part of that, we can see to move the podSecurityContext to the caller function which should the remove the dependency on LinuxContainerConfig without creating WindowsContainerConfig at the pod level

wawa0210 · 2020-08-06T15:50:44Z

Please look at the comment - #93475 (comment). It is my understanding that for the Windows pods there is no need to have sandbox level security Constraints. If we want to push the securityContext outside from Container Security Context to sandbox security constraints it needs a broader discussion as it needs api changes and beyond the scope of current PR

Thank you for your reply. It is true that Windows pods support sandbox level security Constraints and do not belong to the scope of the current pr. But I think this feature is really needed, from #92963 (comment). Because if we need to implement this feature, it will include the changes in the current pr

If possible, we can discuss this topic in another pr or issue #92963 or #93299

liggitt · 2020-08-26T15:58:28Z

this will be moved back into the milestone as part of the phased reopen for 1.20 merges

jsturtevant · 2020-09-03T21:32:14Z

I took a look at this from the perspective of where sig-windows would like to go longer term with the KEP for Privileged Containers and in relation to the issue for supporting runasuser in pod sandbox (#92963)

A few things standout:

As of right not there is no sandbox level security constraints, with Privileged containers that will change. It will be necessary to ensure the user that is running in sandbox and containers have the right privileges to restrict file access and to turn the container into a privileged container.
There is a need to remove the dependency on linux sandbox config (Investigate removal of call to generatePodSandboxLinuxConfig for Windows kubelet #93580)
The e2e tests that is added helps ensure this type of issue doesn't come up again

I think the approach in #93299 is closer to what we need longer term (solves 1 & 2) and would negate the need for the code here that strips the security constraints on the linux configuration but there is more discussion I think that needs to happen (CRI changes and containerd would need to support the sandbox configuration). I also think the test this PR introduces is valuable for sig-windows.

Since the Privileged Container KEP is still in review and the CRI changes are bigger, this LGTM with the caveat that once we split the dependency on linux configuration these checks should be removed (though they wouldn't hurt anything if it was moved to a linux only file as proposed by #93299)

Thoughts?

jsturtevant · 2020-09-03T21:33:20Z

one additional thought, @ravisantoshgudimetla did you look into adding unit tests for this in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_sandbox_test.go?

ravisantoshgudimetla · 2020-09-04T16:59:33Z

Since the Privileged Container KEP is still in review and the CRI changes are bigger, this LGTM with the caveat that once we split the dependency on linux configuration these checks should be removed (though they wouldn't hurt anything if it was moved to a linux only file as proposed by #93299)

@jsturtevant - I'd also like to remove the check on the kubelet side and have the api to have windowsSpecific fields in the long term.

one additional thought, @ravisantoshgudimetla did you look into adding unit tests for this in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_sandbox_test.go?

I initially wanted to add units but there were not many edge cases considering the changes we're making are OS specific.

jsturtevant · 2020-09-10T16:48:36Z

/test pull-kubernetes-e2e-kind-ipv6

jsturtevant · 2020-09-10T16:58:41Z

@SergeyKanzhelev @ddebroy could we get a review/approval?

SergeyKanzhelev · 2020-09-10T20:46:16Z

pkg/kubelet/kuberuntime/kuberuntime_sandbox.go

 	if pod.Spec.SecurityContext != nil {
 		sc := pod.Spec.SecurityContext
-		if sc.RunAsUser != nil {
+		if sc.RunAsUser != nil && runtime.GOOS != "windows" {


it looks like misconfiguration if windows pod configured like this. Should this if has else to report this misconfiguration as a warning?

we discussed this a bit as well at sig-windows, my understanding is the use case where users are running into this is when there was automation that assigns security contexts to all pods. The expectation from the windows perspective since these fields are marked as not supported on windows they would be ignored. @ravisantoshgudimetla can confirm

SergeyKanzhelev · 2020-09-10T20:57:08Z

@ddebroy your approval is needed for test/e2e/windows/OWNERS

ddebroy · 2020-09-10T22:13:14Z

/approve

sjenning · 2020-09-10T22:48:26Z

Issue capturing the weirdness of calling generatePodSandboxLinuxConfig then checking for runtime.GOOS != "windows", which should be implied by the function name, is captured in #93580. We should remedy that.

In the meantime,
/approve

k8s-ci-robot · 2020-09-10T22:48:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ddebroy, ravisantoshgudimetla, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/OWNERS~~ [sjenning]
~~test/e2e/windows/OWNERS~~ [ddebroy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

michmike · 2020-09-10T23:39:30Z

/milestone v1.20

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jul 27, 2020

k8s-ci-robot requested review from brendandburns and dchen1107 July 27, 2020 16:14

sjenning reviewed Jul 27, 2020

View reviewed changes

pkg/securitycontext/util.go Outdated

Copy link

Contributor

sjenning Jul 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra space causing gofmt failure

ddebroy reviewed Jul 27, 2020

View reviewed changes

k8s-ci-robot added this to the v1.20 milestone Jul 28, 2020

jsturtevant reviewed Jul 28, 2020

View reviewed changes

pkg/securitycontext/util.go Outdated

Copy link

Contributor

jsturtevant Jul 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above via: #92355 (comment)

ravisantoshgudimetla force-pushed the fix-kubelet-scc branch from 8531fe8 to b3150b1 Compare July 28, 2020 23:18

k8s-ci-robot added area/kubelet area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jul 28, 2020

ravisantoshgudimetla force-pushed the fix-kubelet-scc branch 3 times, most recently from bcfbf2e to 3bac4eb Compare July 28, 2020 23:29

ravisantoshgudimetla force-pushed the fix-kubelet-scc branch from 3bac4eb to 1f1c448 Compare July 29, 2020 01:29

jsturtevant mentioned this pull request Aug 5, 2020

Feature+Docs: PodSecurityPolicy for Windows #64801

Closed

jsturtevant mentioned this pull request Aug 5, 2020

REQUEST: New membership for jsturtevant kubernetes/org#2083

Closed

6 tasks

liggitt removed this from the v1.20 milestone Aug 26, 2020

ravisantoshgudimetla mentioned this pull request Sep 4, 2020

Windows pods need to support RunAsUserName like Linux pods, not just work containers, not pause containers #92963

Closed

SergeyKanzhelev reviewed Sep 10, 2020

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 10, 2020

k8s-ci-robot added this to the v1.20 milestone Sep 10, 2020

k8s-ci-robot merged commit f5a42d6 into kubernetes:master Sep 11, 2020

SergeyKanzhelev mentioned this pull request Sep 11, 2020

fix windows container root validate #92355

Merged

github-actions bot mentioned this pull request Sep 15, 2020

Week Ending September 13, 2020 dev-obs/actus#223

Open

jsturtevant mentioned this pull request Sep 16, 2020

ingore apparmor on windows #93220

Merged

SergeyKanzhelev mentioned this pull request Jan 21, 2021

Provide RunAsGroup feature for Containers in a Pod kubernetes/enhancements#213

Closed

5 tasks

jsturtevant mentioned this pull request May 11, 2021

PSP Replacement KEP kubernetes/enhancements#2582

Merged

18 tasks

This was referenced Sep 15, 2021

[release-4.6] Rebase 1.19.14 openshift/kubernetes#960

Closed

[release-4.6] Bug 2008266: Rebase 1.19.14 openshift/kubernetes#962

Merged

Strip unnecessary security contexts on Windows #93475

Strip unnecessary security contexts on Windows #93475

Uh oh!

Conversation

ravisantoshgudimetla commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ravisantoshgudimetla commented Jul 27, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddebroy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pjh commented Jul 28, 2020

Uh oh!

yliaog commented Jul 28, 2020

Uh oh!

michmike commented Jul 28, 2020

Uh oh!

jsturtevant Jul 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsturtevant commented Jul 28, 2020

Uh oh!

pjh commented Jul 28, 2020

Uh oh!

ravisantoshgudimetla commented Jul 28, 2020

Uh oh!

ravisantoshgudimetla commented Jul 28, 2020

Uh oh!

ddebroy commented Aug 4, 2020

Uh oh!

wawa0210 commented Aug 5, 2020

Uh oh!

ravisantoshgudimetla commented Aug 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wawa0210 commented Aug 6, 2020

Uh oh!

liggitt commented Aug 26, 2020

Uh oh!

jsturtevant commented Sep 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsturtevant commented Sep 3, 2020

Uh oh!

ravisantoshgudimetla commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsturtevant commented Sep 10, 2020

Uh oh!

jsturtevant commented Sep 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SergeyKanzhelev commented Sep 10, 2020

Uh oh!

ddebroy commented Sep 10, 2020

Uh oh!

sjenning commented Sep 10, 2020

Uh oh!

k8s-ci-robot commented Sep 10, 2020

Uh oh!

michmike commented Sep 10, 2020

Uh oh!

Uh oh!

ravisantoshgudimetla commented Jul 27, 2020 •

edited

Loading

jsturtevant Jul 28, 2020 •

edited

Loading

ravisantoshgudimetla commented Aug 6, 2020 •

edited

Loading

jsturtevant commented Sep 3, 2020 •

edited

Loading

ravisantoshgudimetla commented Sep 4, 2020 •

edited

Loading