api_service: Return cgroup for running pods, add a new filter field to filter pods according to cgroup #2331

yifan-gu · 2016-03-26T01:29:58Z

The last commit is the functional test I added. It:

Run an http server pod on host network.
Check the pod's cgroup
Filter with the pod's cgroup
Terminate the pod
Check the pod's cgroup should be empty.

yifan-gu · 2016-03-26T01:31:02Z

rkt/pods.go

+		// Read the systemd cgroup.
+		// TODO(yifan): What if rkt runs on non-systemd system?
+		// Read the cgroup of 'ID == 1'?
+		if tuples[1] == "name=systemd" {


@alban @iaguis Is this good enough? Or do we have better ideas here?

Using "name=systemd" is ok: we already do this in https://github.com/coreos/rkt/blob/master/stage1/init/init.go#L729

If "name=systemd" does not exist, it should not be fatal: in stage1() we just print a warning. So it works when systemd is not on the host.

I would move getCgroup in cgroup.go, rename it to GetCgroupPathByPid and it can probably share some code with GetOwnCgroupPath().

Then in getBasicPod(), you would just call cgroup.GetCgroupPathByPid(pid, "name=systemd")

This will need to be reworked with unified hierarchy: #1757, but we will see at that time

I agree with @alban 👍

jonboulle · 2016-03-26T12:49:48Z

api/v1alpha/api.proto

        // If not empty, the pods that have all of the annotations will be returned.
        repeated KeyValue annotations = 6;
+
+        // If not empty, the pods whose cgroup are listed will be returned.


Is there a use case for this reverse lookup?

look up by cgroup? Cadvisor as it turns out, especially with systemd run rkt pods, that they can be wherever they want to be (i.e. can have their own slice defined....)

alban · 2016-03-29T13:16:04Z

tests/rkt_api_service_test.go

+	defer conn.Close()
+
+	patches := []string{
+		fmt.Sprintf("--exec=/inspect --serve-http=localhost:%d --serve-http-timeout=%d", testServerPort, testServerTimeout),


Do you use a http server only for synchronization purpose? To stop the app at the correct time?

If so, using --interactive and child.SendLine() like in TestInteractive might be easier.

Then, you would not need to listen on port 3727 on the host net namespace (it might be used by someone else).

cool. will do

alban · 2016-03-30T12:04:15Z

tests/rkt_api_service_test.go

+	}
+
+	if expectedCgroup != p.Cgroup {
+		t.Errorf("Exepcted %q, saw %q", expectedCgroup, p.Cgroup)


s/Exepcted/Expected/

alban · 2016-03-30T12:11:12Z

Semaphore uses Ubuntu 14.04 without systemd and the test TestAPIServiceCroup now fails.

    rkt_api_service_test.go:165: Exepcted
"/machine.slice/machine-rkt\\x2dc01ee887\\x2d538d\\x2d4af3\\x2d8130\\x2d78ae5582f7e5.scope", saw
"/user/1003.user/8.session/machine-rkt\\x2dc01ee887\\x2d538d\\x2d4af3\\x2d8130\\x2d78ae5582f7e5.scope/init.scope"

You could use sd_util.IsRunningSystemd() in checkPod() to detect if systemd is running, and possible use a less strict test with HasSuffix in that case.

alban · 2016-03-30T12:18:18Z

rkt/api_service.go

 	case Running:
 		pod.State = v1alpha.PodState_POD_STATE_RUNNING
 		pod.Networks = getNetworks(p)
+


no need for a new line

yifan-gu · 2016-03-30T16:09:53Z

Semaphore uses Ubuntu 14.04 without systemd and the test TestAPIServiceCroup now fails.

Why the suffix has init.scope ? @alban

alban · 2016-03-30T17:27:06Z

Why the suffix has init.scope ? @alban

systemd moves itself in init.scope since version v226 to support cgroup unified hierarchy. rkt does not support cgroup unified hierarchy yet, but systemd now uses init.scope unconditionally.

See systemd/systemd@efdb023:

This moves PID 1 into a new "init.scope" implicit scope unit in the root
slice. This is necessary since on the unified hierarchy cgroups may
either contain subgroups or processes but not both. PID 1 hence has to
move out of the root cgroup

And NEWS:

A new special scope unit "init.scope" has been introduced
that encapsulates PID 1 of the system. It may be used to
determine resource usage and enforce resource limits on PID
1 itself. PID 1 hence moved out of the root of the control
group tree.

That explains why the tests on Semaphore have different error messages depending whether it is build with the src flavor and systemd-v229, or with the coreos flavor and systemd-225.

I guess the test will have to be smart enough to accept both.

yifan-gu · 2016-03-31T01:58:02Z

I guess the test will have to be smart enough to accept both.

@alban Well, in thit case the response from api service should not contain init.scope because it is expected to return the root cgroup of the pod, right?

So basically:

if systemd version < v226, returns the cgroup of the container's PID1.
if systemd version >= v226, returns the parent cgroup of the container's PID1.

yifan-gu · 2016-03-31T04:54:13Z

But how do we know the version of the systemd inside the pod?

Add Pod.Cgroup to indicate the cgroup of the pod.

If PodFilter.Cgroups is not empty, then only the pods whose cgroups are listed will be returned.

Test the cgroup should be returned for running pods. Test the cgroup filter should work for running pods.

yifan-gu · 2016-03-31T08:27:35Z

@alban Updated the api service to trim any 'init.scope' suffix, also updated the tests

iaguis · 2016-03-31T09:57:37Z

tests/rkt_api_service_test.go

 	}
 }
+
+func TestAPIServiceCroup(t *testing.T) {


s/Croup/Cgroup/

alban · 2016-03-31T10:21:00Z

After fixing the couple of typos, LGTM

alban · 2016-03-31T10:23:23Z

Thanks @yifan-gu! I will merge this and file a following PR for the couple of typos.

Follow up to rkt#2331

api_service: Add logs when inner function fails in ListPods().

3e05c9d

yifan-gu added component/api server area/api labels Mar 26, 2016

yifan-gu added this to the v1.3.0 milestone Mar 26, 2016

yifan-gu added the priority/P1 label Mar 26, 2016

yifan-gu reviewed Mar 26, 2016
View reviewed changes

yifan-gu added the needs/review label Mar 26, 2016

jonboulle reviewed Mar 26, 2016
View reviewed changes

yifan-gu force-pushed the cgroup_path branch from eb883c8 to f38788a Compare March 27, 2016 05:56

yifan-gu assigned jonboulle Mar 29, 2016

alban reviewed Mar 29, 2016
View reviewed changes

jonboulle added needs/some-changes and removed needs/review labels Mar 29, 2016

jonboulle removed their assignment Mar 29, 2016

yifan-gu force-pushed the cgroup_path branch 3 times, most recently from 2c84a59 to 795219b Compare March 29, 2016 19:28

yifan-gu added needs/review and removed needs/some-changes labels Mar 29, 2016

alban reviewed Mar 30, 2016
View reviewed changes

alban added needs/some-changes and removed needs/review labels Mar 30, 2016

yifan-gu force-pushed the cgroup_path branch from 795219b to dbe9cce Compare March 30, 2016 16:12

yifan-gu force-pushed the cgroup_path branch from dbe9cce to 05a7aca Compare March 31, 2016 08:12

yifan-gu added needs/review and removed needs/some-changes labels Mar 31, 2016

Yifan Gu and others added 2 commits March 31, 2016 01:14

api_service: Return cgroup in ListPods()/InspectPod().

fae673f

Add Pod.Cgroup to indicate the cgroup of the pod.

api_service: Add cgroups in the pod filter.

b3a0850

If PodFilter.Cgroups is not empty, then only the pods whose cgroups are listed will be returned.

yifan-gu force-pushed the cgroup_path branch 2 times, most recently from 89a46e8 to 0745eb6 Compare March 31, 2016 08:15

tests: Add tests for api service to test the cgroup.

fa65282

Test the cgroup should be returned for running pods. Test the cgroup filter should work for running pods.

yifan-gu force-pushed the cgroup_path branch from 0745eb6 to fa65282 Compare March 31, 2016 08:25

iaguis reviewed Mar 31, 2016
View reviewed changes

tests/rkt_api_service_test.go

}

}

func TestAPIServiceCroup(t *testing.T) {

Copy link

Member

iaguis Mar 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Croup/Cgroup/

alban merged commit 9322146 into rkt:master Mar 31, 2016

alban added a commit to kinvolk/rkt that referenced this pull request Mar 31, 2016

api_service: fix typos

0976e9e

Follow up to rkt#2331

This was referenced Mar 31, 2016

api_service: fix typos #2347

Merged

TestAPIServiceCgroup fails #2357

Closed

yifan-gu deleted the cgroup_path branch March 31, 2016 17:07

alban mentioned this pull request Apr 5, 2016

No state between prepared and run #2341

Open

Uh oh!

api_service: Return cgroup for running pods, add a new filter field to filter pods according to cgroup #2331

api_service: Return cgroup for running pods, add a new filter field to filter pods according to cgroup #2331

Uh oh!

Conversation

yifan-gu commented Mar 26, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alban commented Mar 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yifan-gu commented Mar 30, 2016

Uh oh!

alban commented Mar 30, 2016

Uh oh!

yifan-gu commented Mar 31, 2016

Uh oh!

yifan-gu commented Mar 31, 2016

Uh oh!

yifan-gu commented Mar 31, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alban commented Mar 31, 2016

Uh oh!

alban commented Mar 31, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants