cgroup: support config cgroup cpu.idle by systemd #23299

wineway · 2022-05-07T12:56:28Z

this patch introduced a new option named cpu.idle in cgroup/cpu, and already added to opencontainers/runtime-spec#1136, seems we should also make systemd could manage it

poettering · 2022-05-09T13:45:19Z

I don't understand what the patch does? the patch seems to add a boolean option, but your patch here adds a numeric option?

so, what does this do? can you explain?

please add an brief explanation of the option to man/systemd.resource-control.xml, where we document all stuff backed by cgroup stuff...

poettering · 2022-05-09T13:46:12Z

/cc @htejun what is this? should we support this?

htejun · 2022-05-09T16:46:06Z

I think it was from google. They wanted hierarchical idle scheduling - ie. at each level, if some cgroups have idle set, they are guaranteed to run only when no non-idle sibling is runnable. It's a niche feature that google and scheduler folks wanted. Maybe leave it be until some people really want it?

kolyshkin · 2022-05-09T17:36:57Z

the patch seems to add a boolean option, but your patch here adds a numeric option?

Quoting the kernel commit message (https://lore.kernel.org/lkml/[email protected]/):

The per-task_group idle field is an integer that currently only holds
either a 0 or a 1. This is explicitly typed as an integer to allow for
further extensions to this API. For example, a negative value may
indicate a highly latency-sensitive cgroup that should be preferred for
preemption/placement/etc.

poettering · 2022-05-10T08:34:54Z

@wineway so is that something you actually intend to use? or something you just wanted to add support for for completeness?

bluca · 2022-05-10T08:49:16Z

@wineway so is that something you actually intend to use? or something you just wanted to add support for for completeness?

Actually I think I could make good use of this setting, sounds like it would nicely complement the Condition PSI settings for long lived services. We could even configure this in background.slice maybe?

poettering · 2022-05-10T08:50:27Z

the patch seems to add a boolean option, but your patch here adds a numeric option?

Quoting the kernel commit message (https://lore.kernel.org/lkml/[email protected]/):

The per-task_group idle field is an integer that currently only holds
either a 0 or a 1. This is explicitly typed as an integer to allow for
further extensions to this API. For example, a negative value may
indicate a highly latency-sensitive cgroup that should be preferred for
preemption/placement/etc.

We don't necessarily want to expose things 1:1 exposing the raw kernel semantics. We genreally try to be friendlier than the kernel. i.e. to me this appears to be something that is boolean for now, and maybe supports more values later, but still a small number. hence maybe expose this conceptually as an enum for now, that happens to also parse a boolean value... We do that for a couple of settings, for example ProtectSystem=, which is primarily a boolean, but also takes some additional, special values.

I am pretty sure exposing a raw integer is not the right approach...

That all said, I'd like to understand what the usecase here is: should we really care? are there people who actually want this? I'd rather avoid adding support for this, unless there's real value for people in this.

bluca · 2022-05-10T10:01:01Z

That all said, I'd like to understand what the usecase here is: should we really care? are there people who actually want this? I'd rather avoid adding support for this, unless there's real value for people in this.

I think I'd use this - for short-lived tasks we now have the ConditionCPUPressure which works nicely, but for long-running services obviously that doesn't help. This could fill that gap.
Another example, we could set this in background.slice, it should fit the definition pretty well.

poettering · 2022-05-10T10:03:09Z

but why wouldn't CPUShares=<somethinglow> not be the better, safer choice for that? totally starving out processes sounds like risky business, no?

wineway · 2022-05-10T10:04:24Z

@wineway so is that something you actually intend to use? or something you just wanted to add support for for completeness?

for this kernel feature, yes, and for the systemd's implementation, for me mainly for the completeness.

we intend to use this kernel feature in container environments to run a regular service and a less latency-sensitive service in one host machine

before this patch merged, many other applications(like kubernetes qos class) use a minimum cpu.weight intend to do the same thing

with this feature, we can use a more explicit configuration and get a better effect.

since mostly container runtimes rely on systemd driver for cgroup, though using cgroupfs directly is an alternative solution, support it for both completeness and the one who uses container with systemd-cgroup driver seems to be worthy

and as @bluca mentioned, this feature is useful besides what is described above.

bluca · 2022-05-10T10:33:29Z

but why wouldn't CPUShares=<somethinglow> not be the better, safer choice for that? totally starving out processes sounds like risky business, no?

Those are hard limits. We use them too, of course - but just like with the conditions, some tasks are of the type 'I'd like to run and use a lot of resources, but it's fine if I don't because of high-prio items'. With a fixed setting you have a hard ceiling regardless of the state of the system.

poettering · 2022-05-10T12:56:20Z

Those are hard limits. We use them too, of course - but just like with the conditions, some tasks are of the type 'I'd like to run and use a lot of resources, but it's fine if I don't because of high-prio items'. With a fixed setting you have a hard ceiling regardless of the state of the system.

i am not sure i grok the "hard limits" and "hard ceiling" references? CPUWeight=1 means "if the system is under CPU pressure give siblings of mine each 100x more CPU than me". (because CPUWeight=100 is the default)

As I understand the cpu.idle thing it means "if the system is under CPU pressure give me nothing".

What concerns me about the cpu.idle thing is that it makes processes totally hang if there's just one runnable task that has idle not set. Since traditionally batch jobs would use CPUWeight=1 or nice=19 or something like this it means that if you run anything like that in parallel then your service doesn't scheduled atall* anymore. and i doubt people would expect that...

bluca · 2022-05-10T13:23:01Z

Those are hard limits. We use them too, of course - but just like with the conditions, some tasks are of the type 'I'd like to run and use a lot of resources, but it's fine if I don't because of high-prio items'. With a fixed setting you have a hard ceiling regardless of the state of the system.

i am not sure i grok the "hard limits" and "hard ceiling" references? CPUWeight=1 means "if the system is under CPU pressure give siblings of mine each 100x more CPU than me". (because CPUWeight=100 is the default)

As I understand the cpu.idle thing it means "if the system is under CPU pressure give me nothing".

What concerns me about the cpu.idle thing is that it makes processes totally hang if there's just one runnable task that has idle not set. Since traditionally batch jobs would use CPUWeight=1 or nice=19 or something like this it means that if you run anything like that in parallel then your service doesn't scheduled _at_all* anymore. and i doubt people would expect that...

Uhm CPUWeight takes into account current pressure? It wasn't obvious from our manpage, but maybe I'm conflating that with CPUQuota in my head

wineway · 2022-05-10T15:57:01Z

What concerns me about the cpu.idle thing is that it makes processes totally hang if there's just one runnable task that has idle not set. Since traditionally batch jobs would use CPUWeight=1 or nice=19 or something like this it means that if you run anything like that in parallel then your service doesn't scheduled atall* anymore. and i doubt people would expect that...

seems the sematic we want is:

Make non-idle processes use full of CPU as if no idle processes are running
Minimize the impact of idle processes on non-idle processes.

in our use case, we have two services service a and serivce b , service a has a dynamic CPU usage and a higher priority, and service b is a batch job that always tries to use all CPU time but we only expect it make full use of remained CPU time and finish in an arbitrary future time.

we benched cpu.idle and minimum cpu.weight, the processes assigned cpu.idle have much less impact on regular processes.

🤔I think it might not a niche requirement, especially in cloud and container environments.

poettering · 2022-05-10T16:20:46Z

I am not against adding this, if people actually intend to use this.

htejun · 2022-05-10T17:36:22Z

I'm not against adding this either but the niche part comes from this being a hierarchical feature. For most use cases, setting SCHED_IDLE is enough. Maybe it'd be more convenient if that can be enforced per cgroup. What the feature implements is idle class per cgroup sibling group. There are use cases this would be useful but the intersection of needing both hierarchical control and the improvements over weight-based control isn't huge. I think we'd be better off with better weight based control (e.g. even widening the dynamic range of weights). So, I'm rather ambivalent. That said, the feature is there and so are the users, so why not?

wineway · 2022-05-11T10:28:16Z

If we adding this, seems we should make CPUIdle an enum value?

CPUIdle=yes
CPUIdle=no
CPUIdle=#something for future extention?

and since we didn't keep the value range the same as the kernel. Should we rename this option?

but I can't find a name better than CPUIdle😂

or we can keep both enum and integer value for it, and convert the enum value to an integer

CPUIdle=yes
CPUIdle=true
# same as CPUIdle=1

CPUIdle=no
CPUIdle=false
CPUIdle=
# same as CPUIdle=0

Werkov · 2022-06-07T18:44:06Z

My 0.02€ -- as systemd adds own abstract layer on top of kernel knobs, let's allow setting CPUWeight=0 (internally translates to cpu.idle attribute). It's a conservative change but it makes user aware that setting weight to 0 is something special, if kernel API is extended in the future, new abstraction (such as CPUIdle=) can be added to systemd too.

wineway · 2022-06-08T03:38:50Z

as systemd adds own abstract layer on top of kernel knobs, let's allow setting CPUWeight=0 (internally translates to cpu.idle attribute). It's a conservative change but it makes user aware that setting weight to 0 is something special, if kernel API is extended in the future, new abstraction (such as CPUIdle=) can be added to systemd too.

@Werkov In my opinion, CPUWeight name same as cpu.weight in cgroup, we had better make them have semantic equally, otherwise it might make confuse, but I'm not sure about this, is there any further discussion?

keszybz · 2022-07-01T16:20:42Z

I think we should support this knob in some fashion. If the kernel added this in cgroups-v2, this means pretty much means we should support it, since we generally try to provide a complete interface for cgroups-v2.

let's allow setting CPUWeight=0

I don't think we should conflate the two. cpu.weight exists also for processes with cpu.idle=1, so we need two switches to configure those independently.

But please make this take a boolean argument in the config. We'll 'yes|no' for now, and if the kernel interface is extended, we can extend the parser. OTOH, internally and in the dbus interface, I think it's fine if we keep this an integer. If kernel allows other values, we'll want to map them 1:1 to our own values.

poettering · 2022-08-10T11:02:17Z

man/systemd.resource-control.xml

+              <listitem>
+                <para>If set to the special string "idle", mark the cgroup for "idle scheduling", which means
+              that it will get CPU resources only when there are no processes not marked in this way to execute in this
+              cgroup or its siblings. This setting corresponds to the <literal>cpu.idle</literal> cgroup attribute.</para>


hmm, this indentation is particularly weird now

👀 fixed here……

poettering · 2022-08-10T11:03:46Z

lgtm, but the indentation of the man page docbook XML is still a bit weird

man/systemd.resource-control.xml

Signed-off-by: wineway <[email protected]>

Systemd v252 (available in CentOS Stream 9 in our CI) added support for setting cpu.idle (see [1]). The way it works is: - if CPUWeight == 0, cpu.idle is set to 1; - if CPUWeight != 0, cpu.idle is set to 0. This behavior breaks the existing test case, as described in [2]. To fix, skip the last check in the test case in case a newer systemd is used. Fixes: opencontainers#3786 [1] systemd/systemd#23299 [2] opencontainers#3786 Signed-off-by: Kir Kolyshkin <[email protected]>

Systemd v252 (available in CentOS Stream 9 in our CI) added support for setting cpu.idle (see [1]). The way it works is: - if CPUWeight == 0, cpu.idle is set to 1; - if CPUWeight != 0, cpu.idle is set to 0. This commit implements setting cpu.idle in systemd cgroup driver via a unit property. In case CPUIdle is set to non-zero value, the driver sets adds CPUWeight=0 property, which will result in systemd setting cpu.idle to 1. Unfortunately, there's no way to set cpu.idle to 0 without also changing the CPUWeight value, so the driver doesn't do anything if CPUIdle is explicitly set to 0. This case is handled by the fs driver which is always used as a followup to setting systemd unit properties. Also, handle cpu.idle set via unified map. In case it is set to non-zero value, add CPUWeight=0 property, and ignore cpu.weight (otherwise we'll get two different CPUWeight properties set). Add a unit test for new values in unified map, and an integration test case. [1] systemd/systemd#23299 [2] opencontainers#3786 Signed-off-by: Kir Kolyshkin <[email protected]>