Fix ExecSync support for runtimes other than runC #310

sameo · 2017-01-11T09:33:51Z

Some OCI container runtimes (in particular the hypervisor based ones like e.g. Clear Containers) will typically create a shim process between the hypervisor and the runtime caller, in order to not rely on the hypervisor process for e.g. forwarding the output streams or getting a command exit code.

When executing a command inside a running container those runtimes will create that shim process and terminate.
Therefore calling and monitoring the runtime process directly from ExecSync() will fail. Instead we need to have a subreaper calling the runtime and monitoring the shim process, and conmon seems to be a natural place for doing so.

This PR does mostly 2 things:

Add a -e option to conmon to also monitor potential shim processes created by <runtime> exec
Modify ExecSync to call conmon instead of calling the runtime directly. ExecSync also now reads the exit status code back through the synchronization pipe.

Fixes #309

runcom · 2017-01-11T09:58:13Z

I'm overall ok to have conmon exec, @mrunalp @cyphar PTAL

runcom · 2017-01-11T09:55:27Z

test/ctr.bats

 	run ocic ctr execsync --id "$ctr_id" doesnotexist
 	echo "$output"
 	[ "$status" -ne 0 ]
-	[[ "$output" =~ "executable file not found in" ]]


Where has this gone?

This string was returned by runC itself. Other runtimes may not decide to return the same string.
To be runtime agnostic, I think the only thing we can safely check here is that this execsync call really failed.

runcom · 2017-01-11T09:56:16Z

oci/oci.go

 // ExecSync execs a command in a container and returns it's stdout, stderr and return code.
 func (r *Runtime) ExecSync(c *Container, command []string, timeout int64) (resp *ExecSyncResponse, err error) {
-	args := []string{"exec", c.name}
+	parentPipe, childPipe, err := os.Pipe()


I'd create a function in the oci package to wrap all of this? What do you think?

By "all of this" do you mean the pipe and temporary file creation ?

By "all of this" do you mean the pipe and temporary file creation ?

yes

laijs · 2017-01-11T12:03:15Z

When executing a command inside a running container those runtimes will create that shim process and terminate.

I don't understand it, if the argument "-d" is not added to "runtime exec", the runtime exec process must wait the new process of the container.

sameo · 2017-01-11T13:11:32Z

@laijs You're correct. I'll try playing with that. It does not work currently with clear containers, but it may be a simpler path.

laijs · 2017-01-11T13:24:06Z

I like this approach also, but I think you need to add "-d" to "runtime exec" in ~~ExecSync()~~ common.c

sameo · 2017-01-11T14:32:04Z

@laijs I assume you meant adding -d to the conmon exec arguments ?

sameo · 2017-01-11T16:17:33Z

@laijs Done, conmon now calls exec with -d.

mrunalp · 2017-01-12T15:35:27Z

Thanks! I will test this out today.

mrunalp · 2017-01-13T19:27:11Z

conmon/conmon.c

 	/* Wait for the container process and record its exit code */
 	while ((pid = waitpid(-1, &status, 0)) > 0) {
-		printf("PID %d exited\n", pid);
+		int exit_status = WEXITSTATUS(status);


We were doing this conversion up in the go code so need to remove the conversion from https://github.com/kubernetes-incubator/cri-o/blob/master/oci/oci.go#L338

Done, thanks for spotting that one.

waitpid fills its second argument with a value that contains the process exit code in the 8 least significant bits. Instead of returning the complete value and then convert it from ocid, return the exit status directly by using WEXITSTATUS from conmon. Signed-off-by: Samuel Ortiz <[email protected]>

And not a hardcoded "pidfile". Signed-off-by: Samuel Ortiz <[email protected]>

Some OCI container runtimes (in particular the hypervisor based ones) will typically create a shim process between the hypervisor and the runtime caller, in order to not rely on the hypervisor process for e.g. forwarding the output streams or getting a command exit code. With these runtimes we need to monitor a different process than the runtime one when executing a command inside a running container. The natural place to do so is conmon and thus we add a new option to conmon for calling the runtime exec command, monitor the PID and then return the running command exit code through the sync pipe to the parent. Signed-off-by: Samuel Ortiz <[email protected]>

Some OCI container runtimes (in particular the hypervisor based ones) will typically create a shim process between the hypervisor and the runtime caller, in order to not rely on the hypervisor process for e.g. forwarding the output streams or getting a command exit code. When executing a command inside a running container those runtimes will create that shim process and terminate. Therefore calling and monitoring them directly from ExecSync() will fail. Instead we need to have a subreaper calling the runtime and monitoring the shim process. This change uses conmon as the subreaper from ExecSync(), monitors the shim process and read the exec'ed command exit code from the synchronization pipe. Signed-off-by: Samuel Ortiz <[email protected]>

"executable file not found in" is part of a runc specific output when 'runc exec' fails. This prevents the execsync failure to pass when running ocid with other runtimes than runc. Signed-off-by: Samuel Ortiz <[email protected]>

mrunalp · 2017-01-16T19:41:09Z

LGTM

cyphar · 2017-01-24T08:30:58Z

Sorry that I was on holiday while this patch was being reviewed. I don't like this because now all of conmon has bifoccated logic for exec and non-exec. The code is getting really hard to follow...

cyphar · 2017-01-24T17:21:21Z

I'll do a bit of cleanup in #162 ...

sameo · 2017-01-24T17:26:46Z

@cyphar Sounds good. Please ping me when you have it ready so that I can verify it does not break Clear Containers.

k8s-ci-robot added the cncf-cla: yes label Jan 11, 2017

sameo force-pushed the topic/cc-exec branch 2 times, most recently from 391b70f to 9fd2477 Compare January 11, 2017 09:52

runcom reviewed Jan 11, 2017

View reviewed changes

sameo force-pushed the topic/cc-exec branch from 9fd2477 to 15abcfc Compare January 11, 2017 16:04

mrunalp reviewed Jan 13, 2017

View reviewed changes

Samuel Ortiz added 5 commits January 14, 2017 02:00

conmon: Use the full PID file path

468746a

And not a hardcoded "pidfile". Signed-off-by: Samuel Ortiz <[email protected]>

test: Do not hardcode runc specific output

ce54c1e

"executable file not found in" is part of a runc specific output when 'runc exec' fails. This prevents the execsync failure to pass when running ocid with other runtimes than runc. Signed-off-by: Samuel Ortiz <[email protected]>

sameo force-pushed the topic/cc-exec branch from 15abcfc to ce54c1e Compare January 14, 2017 01:04

mrunalp merged commit 2421aba into cri-o:master Jan 16, 2017

haircommander mentioned this pull request Jul 20, 2021

oci: properly handle tty on execsync #5107

Merged

Fix ExecSync support for runtimes other than runC #310

Fix ExecSync support for runtimes other than runC #310

Uh oh!

Conversation

sameo commented Jan 11, 2017

Uh oh!

runcom commented Jan 11, 2017

Uh oh!

runcom Jan 11, 2017

Choose a reason for hiding this comment

Uh oh!

sameo Jan 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

runcom Jan 11, 2017

Choose a reason for hiding this comment

Uh oh!

sameo Jan 11, 2017

Choose a reason for hiding this comment

Uh oh!

runcom Jan 11, 2017

Choose a reason for hiding this comment

Uh oh!

laijs commented Jan 11, 2017

Uh oh!

sameo commented Jan 11, 2017

Uh oh!

laijs commented Jan 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sameo commented Jan 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sameo commented Jan 11, 2017

Uh oh!

mrunalp commented Jan 12, 2017

Uh oh!

mrunalp Jan 13, 2017

Choose a reason for hiding this comment

Uh oh!

sameo Jan 14, 2017

Choose a reason for hiding this comment

Uh oh!

mrunalp commented Jan 16, 2017

Uh oh!

cyphar commented Jan 24, 2017

Uh oh!

cyphar commented Jan 24, 2017

Uh oh!

sameo commented Jan 24, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sameo Jan 11, 2017 •

edited

Loading

laijs commented Jan 11, 2017 •

edited

Loading

sameo commented Jan 11, 2017 •

edited

Loading