Add option to drop infra container #4165

haircommander · 2020-09-04T17:40:41Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

 Sandbox: Don't use an infra container in some cases

some changes this includes:
- allow for the infra container to not be created by the runtime, if sb.NeedsInfra() is false
- add concept of a 'spoofed' container. This allows us to make calls to ID(), Name(), and Description() without having the runtime create the container
- using spoof, change checks of podInfraContainer == nil to Spoofed(), as well as re-add Description() calls
- reorder some infra container creation code
- add unit tests for NeedsInfra and Spoofed
- add integration test for drop infra
- sysctls are configured in the runtime after the process unshares in the shared namespaces. since we aren't using a runtime, we need to repeat this step.
        - After unsharing, but before binding, configure the sysctls for the namespace.
- spoof container state
- create a new annotations package for cri-o specific annotations that are experimental, and not yet locked in the API
- refactor ContainerRuntimeType to get a runtime type without a container (and call it RuntimeType)

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

add `--drop-infra-ctr` option to ask CRI-O to drop the infra container when a pod level pid namespace isn't requested. This feature is considered experimental

haircommander · 2020-09-04T17:41:10Z

replaces #3997 (which has grown out of date, and has an extra patch that's not needed)

haircommander · 2020-09-04T17:42:46Z

also carries #4146

haircommander · 2020-09-04T17:44:10Z

/retest

codecov · 2020-09-04T20:30:24Z

Codecov Report

Merging #4165 into master will decrease coverage by 0.00%.
The diff coverage is 40.95%.

@@            Coverage Diff             @@
##           master    #4165      +/-   ##
==========================================
- Coverage   40.92%   40.91%   -0.01%     
==========================================
  Files         111      111              
  Lines        9512     9614     +102     
==========================================
+ Hits         3893     3934      +41     
- Misses       5242     5301      +59     
- Partials      377      379       +2

mrunalp · 2020-09-08T14:59:35Z

internal/criocli/criocli.go

 	if ctx.IsSet("manage-ns-lifecycle") {
 		config.ManageNSLifecycle = ctx.Bool("manage-ns-lifecycle")
 	}
+	if ctx.IsSet("drop-infra") {


Wondering if calling this drop-infra-container or drop-infra-ctr is better or leave it as is.

yeah drop-infra-ctr is more clear... I may make that change on next rebase (the only other field in the config that has container or ctr is ctr_stop_timeout so we should keep with ctr imo)

haircommander · 2020-09-08T19:34:41Z

/retest

mrunalp · 2020-09-08T20:29:46Z

internal/criocli/criocli.go

 		},
+		&cli.BoolFlag{
+			Name:    "drop-infra-ctr",
+			Usage:   fmt.Sprintf("determines whether we drop the infra container when appropriate. Requires ManageNSLifecycle to be true (default: %v)", defConf.DropInfraCtr),


instead of when appropriate should we reword to when pod pid namespace is not shared?

I thought it a bit verbose for a cli flag TBH but yeah it would better match the config values

mrunalp · 2020-09-08T20:39:35Z

internal/lib/container_server.go

 // on disk
 func (c *ContainerServer) ContainerStateToDisk(ctr *oci.Container) error {
-	if ctr == nil {
+	if ctr.Spoofed() {


Do we need a skip in ContainerStateFromDisk as well?

we can for posterity, but the only time it's called is when restoring, in a block where we don't restore if the container used to be spoofed.

internal/lib/sandbox/namespaces_linux.go

mrunalp · 2020-09-08T20:43:18Z

internal/lib/sandbox/sandbox.go

 	return nil
 }
+
+// NeedsInfra is a function that returns whether the sandbox will need an infra container


nit: Add sentence with period.

internal/lib/sandbox/sandbox.go

mrunalp · 2020-09-08T20:46:50Z

internal/lib/sandbox/sandbox.go

+
+// NeedsInfra is a function that returns whether the sandbox will need an infra container
+// if the server manages the namespace lifecycles, and the Pid option on the sandbox
+// is node level, the infra container is not needed


node or container

internal/oci/container.go

internal/lib/sandbox/sandbox.go

Signed-off-by: Peter Hunt <[email protected]>

as well as update existing tests to allow drop_infra_ctr to be true or false by default Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2020-09-11T01:14:33Z

(most) comments address, thanks for the thorough review @kolyshkin

PTAL @umohnani8 @mrunalp @saschagrunert

specifically, set RUNTIME_NAME and CONTAINER_DEFAULT_RUNTIME if it's set as such Signed-off-by: Peter Hunt <[email protected]>

sysctls must be passed to pinns of the format -s 'key=value'+'key2=value2' also, move the pinns source code to `pinns/src` to clean up the top level directory a bit Signed-off-by: Peter Hunt <[email protected]>

as it would otherwise unconditionally have pexit() called Signed-off-by: Peter Hunt <[email protected]>

Signed-off-by: Peter Hunt <[email protected]>

to hold crio specific annotations until they are stable enough to put in containers/common Signed-off-by: Peter Hunt <[email protected]>

as well as a constructor for a spoofed container a Spoofed() container is one that exists in cri-o's state, but not in the runtime This allows us to drop the infra, but keep bookkeeping standardized (for instance, when we restore cri-o, we look at the container's state directory) Signed-off-by: Peter Hunt <[email protected]>

to allow the sandbox to decide when it needs an infra container Signed-off-by: Peter Hunt <[email protected]>

in the initial (very old) implementation of a dropped infra, infra containers were allowed to be nil This is no longer the case, so we can remove traces of that old implementation Signed-off-by: Peter Hunt <[email protected]>

this commit includes: - propegate sysctls from pod creation down to pinns - use the annotations package for userns and spoofed containers - add restore code for when a container was spoofed - query a runtime handler based on the string instead of by container - this allows us to query the runtime handler before we create the container to allow it to be spoofed - drop the infra container when the pod is not kernel separated, and when NeedsInfra is false Signed-off-by: Peter Hunt <[email protected]>

saschagrunert

Just one suggestion around the general behavior, otherwise LGTM

saschagrunert · 2020-09-11T08:45:48Z

docs/crio.8.md


 **--default-ulimits**="": Ulimits to apply to containers by default (name=soft:hard) (default: []) (default: [])

+**--drop-infra-ctr**: Determines whether pods are created without an infra container (when the pod is not using a pod level PID namespace). Requires ManageNSLifecycle to be true (default: false)


I suggest we implicitly set ManageNSLifecycle to true if this toggle switches on.

I think it is more clear (as an experimental feature) that we fail quickly when it's not set. By the time DropInfra is set to true by default, I like to think we won't have ManageNSLifecycle as an option anymore

openshift-ci-robot · 2020-09-11T08:51:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2020-09-11T17:35:56Z

/retest

mrunalp · 2020-09-11T18:27:24Z

/lgtm

haircommander · 2020-09-12T04:41:57Z

/retest

openshift-ci-robot · 2020-09-12T05:16:36Z

@haircommander: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2	`9afdd35`	link	`/test e2e_cgroupv2`
ci/prow/e2e-aws	`9afdd35`	link	`/test e2e-aws`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Add option to drop infra container Signed-off-by: bin liu <[email protected]>

haircommander requested review from mrunalp and runcom as code owners September 4, 2020 17:40

openshift-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Sep 4, 2020

openshift-ci-robot requested review from nalind and rhatdan September 4, 2020 17:40

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 4, 2020

haircommander force-pushed the drop-infra branch 2 times, most recently from 9a0509a to 53c5bd4 Compare September 4, 2020 19:57

haircommander force-pushed the drop-infra branch 2 times, most recently from b68bade to 25681e1 Compare September 4, 2020 22:32

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 7, 2020

haircommander force-pushed the drop-infra branch from 25681e1 to 90975f3 Compare September 8, 2020 14:37

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 8, 2020

mrunalp reviewed Sep 8, 2020

View reviewed changes

This was referenced Sep 8, 2020

crio: use drop-in file to configure kata runtime kata-containers/tests#2834

Merged

crio tests are configured to use runc, not kata kata-containers/tests#2835

Closed

haircommander force-pushed the drop-infra branch 2 times, most recently from 4145bbc to a3787af Compare September 8, 2020 18:03

mrunalp reviewed Sep 8, 2020

View reviewed changes

internal/lib/sandbox/namespaces_linux.go Show resolved Hide resolved

mrunalp reviewed Sep 8, 2020

View reviewed changes

internal/lib/sandbox/sandbox.go Outdated Show resolved Hide resolved

mrunalp reviewed Sep 8, 2020

View reviewed changes

kolyshkin reviewed Sep 10, 2020

View reviewed changes

internal/oci/container.go Outdated Show resolved Hide resolved

kolyshkin reviewed Sep 10, 2020

View reviewed changes

internal/oci/container.go Outdated Show resolved Hide resolved

kolyshkin reviewed Sep 10, 2020

View reviewed changes

internal/oci/container.go Outdated Show resolved Hide resolved

kolyshkin reviewed Sep 10, 2020

View reviewed changes

internal/lib/sandbox/sandbox.go Outdated Show resolved Hide resolved

haircommander added 2 commits September 10, 2020 20:52

add --drop-infra-ctr option

3978b8c

Signed-off-by: Peter Hunt <[email protected]>

test: add tests for dropping infra

9e337b0

as well as update existing tests to allow drop_infra_ctr to be true or false by default Signed-off-by: Peter Hunt <[email protected]>

haircommander force-pushed the drop-infra branch from e6357c8 to 75b5f18 Compare September 11, 2020 01:14

haircommander added 9 commits September 10, 2020 22:26

test: configure self when kata-runtime is the CONTAINER_RUNTIME

3f655aa

specifically, set RUNTIME_NAME and CONTAINER_DEFAULT_RUNTIME if it's set as such Signed-off-by: Peter Hunt <[email protected]>

pinns: allow sysctls to be passed

26de5b6

sysctls must be passed to pinns of the format -s 'key=value'+'key2=value2' also, move the pinns source code to `pinns/src` to clean up the top level directory a bit Signed-off-by: Peter Hunt <[email protected]>

pinns: fix pinning cgroup namespace

3c241bd

as it would otherwise unconditionally have pexit() called Signed-off-by: Peter Hunt <[email protected]>

portforward: rework to not need infra container

4ff61bb

Signed-off-by: Peter Hunt <[email protected]>

introduce pkg/annotations

33de444

to hold crio specific annotations until they are stable enough to put in containers/common Signed-off-by: Peter Hunt <[email protected]>

sandbox: add NeedsInfra function

e42b2b1

to allow the sandbox to decide when it needs an infra container Signed-off-by: Peter Hunt <[email protected]>

server: no longer assume some infra containers will be nil

25383e7

in the initial (very old) implementation of a dropped infra, infra containers were allowed to be nil This is no longer the case, so we can remove traces of that old implementation Signed-off-by: Peter Hunt <[email protected]>

haircommander force-pushed the drop-infra branch from 75b5f18 to 9afdd35 Compare September 11, 2020 02:26

saschagrunert approved these changes Sep 11, 2020

View reviewed changes

openshift-ci-robot assigned mrunalp Sep 11, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 11, 2020

haircommander changed the title ~~Drop infra container, again again~~ Add option to drop infra container Sep 11, 2020

openshift-merge-robot merged commit a001e72 into cri-o:master Sep 12, 2020

liubin pushed a commit to liubin/cri-o that referenced this pull request Sep 16, 2020

Merge pull request cri-o#4165 from haircommander/drop-infra

6261ff8

Add option to drop infra container Signed-off-by: bin liu <[email protected]>

haircommander mentioned this pull request Oct 13, 2020

WIP: drop the infra container (again!) #3997

Closed


		--default-ulimits="": Ulimits to apply to containers by default (name=soft:hard) (default: []) (default: [])

		--drop-infra-ctr: Determines whether pods are created without an infra container (when the pod is not using a pod level PID namespace). Requires ManageNSLifecycle to be true (default: false)

Add option to drop infra container #4165

Add option to drop infra container #4165

Uh oh!

Conversation

haircommander commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

haircommander commented Sep 4, 2020

Uh oh!

haircommander commented Sep 4, 2020

Uh oh!

haircommander commented Sep 4, 2020

Uh oh!

codecov bot commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Sep 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haircommander commented Sep 11, 2020

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Sep 11, 2020

Uh oh!

haircommander commented Sep 11, 2020

Uh oh!

mrunalp commented Sep 11, 2020

Uh oh!

haircommander commented Sep 12, 2020

Uh oh!

openshift-ci-robot commented Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

haircommander commented Sep 4, 2020 •

edited

Loading

codecov bot commented Sep 4, 2020 •

edited

Loading

openshift-ci-robot commented Sep 12, 2020 •

edited

Loading