oci: check pid before using it #3868

haircommander · 2020-06-11T21:13:26Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

As we found in the conmonmon saga, directly accessing pids on the host is a bit dangerous. Primarily, we risk believing a pid is the container process, when it is really a new process, because of pid wrap. While this is really unlikely to happen if you set it to the kernel pid_max (4 billion?), it still can happen, and can cause security problems (especially when using it for namespaces, when not managing namespace lifecycle)

Now, we drop every direct c.state.Pid access in favor of a helper function c.Pid() (and children). This allows us to do the proper verification (checking stime is the same for the pid we're trying to use as the container's pid originally was)

This PR also refactors how we use FindProcess(), as there was a bit of duplication, and direct accesses to c.state.Pid were littered in code associated with FindProcess()

This also carries the exec sync patch checking c.IsRunning() instead of updating container state, as a bonus.

TODO:

test restore case (I need to find the best way to persist the pid to disk)
add more tests
~~- [ ] maybe add this pid handling stuff in a new package for better unit testing~~

But I'm throwing this up to see how CI likes it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

haircommander · 2020-06-11T21:15:24Z

/retest

mrunalp · 2020-06-11T21:16:58Z

internal/oci/container.go

+	// the unix start time of the process, as found by psgo
+	// this is used to track whether the pid we have stored
+	// is the same as the same pid number on the host
+	psStartTime        string


How about pidStartTime ?

yeah that's better. I was at a loss for a variable name :)

processStartTime is probably better than that.

This will have to be persisted in state.

I am thinking that we should probably separate the pid returned from runc state from the container start pid. runc state could change to zero once stopped. We can separately store the startTime, Pid as top level fields in crio state that never change once populated.

internal/oci/container.go

server/container_create_linux.go

haircommander · 2020-06-17T14:53:39Z

/retest

codecov · 2020-06-17T15:18:08Z

Codecov Report

Merging #3868 into master will increase coverage by 0.29%.
The diff coverage is 64.70%.

@@            Coverage Diff             @@
##           master    #3868      +/-   ##
==========================================
+ Coverage   40.41%   40.71%   +0.29%     
==========================================
  Files         111      109       -2     
  Lines        8947     8970      +23     
==========================================
+ Hits         3616     3652      +36     
+ Misses       4999     4986      -13     
  Partials      332      332

haircommander · 2020-06-17T19:32:12Z

/retest

haircommander · 2020-06-17T19:55:17Z

/retest

haircommander · 2020-06-17T20:45:28Z

integration tests are a runc problem fixed in opencontainers/runc#2479, e2e test failures are legit

haircommander · 2020-06-19T19:41:25Z

/retest

haircommander · 2020-07-30T14:09:49Z

/retest

kolyshkin · 2020-07-30T18:52:29Z

internal/oci/container.go

+}
+
+// getInitStartTime reads the kernel's /proc entry for stime for PID.
+func getInitStartTime(pid int) (int, error) {


The word Init in the function name is misleading, since the function doesn't care if the pid is that of init or not.

mrunalp · 2020-07-30T18:53:42Z

internal/oci/oci_linux.go

+	// won't exist anyway.
+	pid, _ := ctr.pid() // nolint:errcheck
+	if pid > 0 {
+		netNsPath := fmt.Sprintf("/proc/%d/ns/net", pid)


Why don't we return the pinned path here? We can get rid of the pid check then.

a couple of reasons:

primarily: we don't have direct access to the Sandbox object, which owns the namespaces. We'd have to ask the server what the sandbox is, and then ask the sandbox what the namespace is.
secondarily, is it possible to have a private net namespace? if so, the sandbox level namespace won't be the correct one

Atleast for the pinned ns case.

There are no private net namespaces in a pod. All the containers always share the pod network namespace.

ah, well we still have the access problem. I would replumb this as a follow up, if we'd like

having exec sync update state each time is a bit excessive. In addition to exec'ing extra, it causes potential for runc state to flake, causing the container to go down. instead, we should just check if the pid is running, and proceed if so Signed-off-by: Peter Hunt <[email protected]>

in any case where we want to directly manipulate a pid on the host. this is unsafe, as we can encounter pid wrap. for those cases, we need to check the pid is the one we want to access, whether it's creating a namespace path with /proc or directly killing a container also, add unit tests for this fix, as well as refactor a few of the oci tests Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2020-07-30T20:57:11Z

@kolyshkin pointed out that most of findprocess is now obsolete with the existence of verifyPid(). I have removed it and associated calls

now, verifyPid does all of the juicy bits of findprocess, without any of the overhead. Further, it added a lot of code that was largely written as a compatibility layer for windows. Since we have no intention in the near future to support windows anymore, let's drop it Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2020-07-31T02:01:34Z

/retest

TomSweeneyRedHat · 2020-07-31T13:25:56Z

internal/lib/sandbox/namespaces.go

+	// is valid. If not, infraPid should be less than or equal to 0
 	if ns == nil || ns.Get() == nil {
-		if infraPid >= 0 {
+		if infraPid > 0 {


this doesn't match the new comment on 319. Did you mean to remove the = here?

the comment is meant to say "if it's not valid". is the wording unclear?

I also had to read this part twice, and it is correct, including the comment, but maybe not super clear. Not sure how to improve it though :(

kolyshkin · 2020-07-30T20:14:39Z

internal/oci/container.go

+		return false, nil
+	}
+
+	process, err := findprocess.FindProcess(pid)


I was not suggesting to drop process.Release(), but drop both find and release. We have already checked the process is there, I'm not sure if it makes sense to do it one more time.

kolyshkin · 2020-07-31T19:12:50Z

internal/lib/sandbox/namespaces.go

+	// is valid. If not, infraPid should be less than or equal to 0
 	if ns == nil || ns.Get() == nil {
-		if infraPid >= 0 {
+		if infraPid > 0 {


I also had to read this part twice, and it is correct, including the comment, but maybe not super clear. Not sure how to improve it though :(

kolyshkin · 2020-07-31T19:24:58Z

server/container_execsync.go

-
-	cState := c.State()
-	if !(cState.Status == oci.ContainerStateRunning || cState.Status == oci.ContainerStateCreated) {
+	if !c.IsAlive() {


maybe

if _, err := c.verifyPid(); err != nil {

and remove the whole IsAlive function?

Or, use IsAlive in all the places where you're using the if like the one in this comment. Otherwise it's a duplicated functionality.

the nice thing about the different functions has to do with error handling. In the case of IsAlive, I believe it should be used in cases we presume the container to be alive, and it would be an error if it was not.

Conversely, verifyPid does not make an assertion of what the liveliness of the container should be

For instance, all uses in runtime_oci.go assume the container will eventually not be alive. in that case, we wouldn't want to print an error that the container wasn't alive

I think it is correct to error in server/container_execsync.go, as the client asked to exec in a container that is not running. Propagating the reason for this failure (pid wrap, container exited, we exec'ed too early somehow) is useful.

Do you have suggestions on how to balance these two use cases and error scenerios @kolyshkin

haircommander · 2020-08-07T17:48:05Z

/retest

mrunalp · 2020-08-07T21:53:27Z

/test e2e-aws

haircommander · 2020-08-10T13:12:38Z

/retest

mrunalp · 2020-08-10T14:45:59Z

/lgtm

haircommander · 2020-08-10T15:06:32Z

/retest

haircommander · 2020-08-10T17:15:06Z

/retest

haircommander · 2020-08-10T19:22:29Z

/retest

haircommander · 2020-08-11T12:58:17Z

/retest

haircommander requested review from mrunalp and runcom as code owners June 11, 2020 21:13

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 11, 2020

openshift-ci-robot requested review from giuseppe, nalind and umohnani8 June 11, 2020 21:13

openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 11, 2020

mrunalp reviewed Jun 11, 2020

View reviewed changes

haircommander mentioned this pull request Jun 11, 2020

make all operations read state atomically #3864

Closed

mrunalp reviewed Jun 11, 2020

View reviewed changes

internal/oci/container.go Outdated Show resolved Hide resolved

haircommander commented Jun 11, 2020

View reviewed changes

server/container_create_linux.go Outdated Show resolved Hide resolved

haircommander mentioned this pull request Jun 17, 2020

reduce flakiness of UpdateContainerStatus #3867

Closed

haircommander force-pushed the check-pid branch from 6de799e to 0f16272 Compare June 17, 2020 14:50

haircommander force-pushed the check-pid branch 2 times, most recently from fc5fabe to 641d4fe Compare June 17, 2020 18:53

haircommander changed the title ~~WIP oci: check pid before using it~~ oci: check pid before using it Jun 17, 2020

openshift-ci-robot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 17, 2020

haircommander force-pushed the check-pid branch from 641d4fe to e269ff3 Compare June 17, 2020 19:42

haircommander force-pushed the check-pid branch 2 times, most recently from e9e10bf to eaeae76 Compare June 19, 2020 16:41

kolyshkin reviewed Jul 30, 2020

View reviewed changes

mrunalp reviewed Jul 30, 2020

View reviewed changes

haircommander force-pushed the check-pid branch from a09d8d8 to 7788c4e Compare July 30, 2020 18:59

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 30, 2020

haircommander force-pushed the check-pid branch from 7788c4e to c20e72c Compare July 30, 2020 19:46

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 30, 2020

haircommander force-pushed the check-pid branch 2 times, most recently from 9c32a5e to ee7a9a0 Compare July 30, 2020 20:56

haircommander force-pushed the check-pid branch from ee7a9a0 to 3344d25 Compare July 30, 2020 21:00

haircommander force-pushed the check-pid branch from 3344d25 to 44e0c0d Compare July 30, 2020 21:05

TomSweeneyRedHat reviewed Jul 31, 2020

View reviewed changes

kolyshkin requested changes Jul 31, 2020

View reviewed changes

openshift-ci-robot assigned mrunalp Aug 10, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2020

haircommander added the release-1.19 label Aug 10, 2020

openshift-merge-robot merged commit 7184c0f into cri-o:master Aug 11, 2020

oci: check pid before using it #3868

oci: check pid before using it #3868

Conversation

haircommander commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

haircommander commented Jun 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrunalp Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haircommander commented Jun 17, 2020

Uh oh!

codecov bot commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haircommander commented Jun 17, 2020

Uh oh!

haircommander commented Jun 17, 2020

Uh oh!

haircommander commented Jun 17, 2020

Uh oh!

haircommander commented Jun 19, 2020

Uh oh!

haircommander commented Jul 30, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrunalp Jul 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jul 30, 2020

Uh oh!

haircommander commented Jul 31, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Aug 7, 2020

Uh oh!

mrunalp commented Aug 7, 2020

Uh oh!

haircommander commented Aug 10, 2020

haircommander commented Jun 11, 2020 •

edited

Loading

mrunalp Jun 11, 2020 •

edited

Loading

codecov bot commented Jun 17, 2020 •

edited

Loading

mrunalp Jul 30, 2020 •

edited

Loading