internal/process: add functionality to clean up zombie children #5260

haircommander · 2021-08-27T16:28:23Z

Now that exec sync requests are run by conmon, there are more processes in the mix and more possibility for zombies

this commit adds a zombie monitor to the defunct process metrics collection flow. it is a little clunky, but it would be weird
to have two different /proc parsers for very similar uses

Signed-off-by: Peter Hunt [email protected]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

add a zombie monitor to cleanup CRI-O's defunct children

codecov · 2021-08-27T16:32:37Z

Codecov Report

Merging #5260 (d2d5608) into master (35d1be8) will increase coverage by 0.09%.
The diff coverage is 67.92%.

❗ Current head d2d5608 differs from pull request most recent head 4bdcabb. Consider uploading reports for the commit 4bdcabb to get more accurate results

@@            Coverage Diff             @@
##           master    #5260      +/-   ##
==========================================
+ Coverage   43.76%   43.86%   +0.09%     
==========================================
  Files         118      119       +1     
  Lines       11721    11757      +36     
==========================================
+ Hits         5130     5157      +27     
- Misses       6104     6112       +8     
- Partials      487      488       +1

saschagrunert · 2021-08-30T08:20:29Z

internal/process/defunct_processes.go

+// ParseDefunctProcessesForPath retrieves the number of zombie processes from
+// a specific process filesystem, as well as the number of defunct children of the current running process.
+func ParseDefunctProcessesForPathAndParent(path string, parent int) (defunctCount uint, defunctChildren []int, retErr error) {
+	defunctChildren = make([]int, 0)


This line seems not required

Suggested change

defunctChildren = make([]int, 0)

saschagrunert · 2021-08-30T08:22:37Z

internal/process/zombie_monitor.go

+// It fails if one has already been created.
+func InitializeZombieMonitor() error {
+	if monitor != nil {
+		return errors.New("zombie monitor already intialized")


I know this will probably not happen, but we could return the existing monitor here.

saschagrunert · 2021-08-30T08:26:28Z

server/metrics/metrics.go

-				}
-				logrus.Warn(err)
-				return 0
+				return float64(process.TotalDefunctProcesses())


This makes the "return the current state of the system" call time-wise dependent on the zombie monitor (runs every 5 seconds for now). I'm not sure if we wanna have this coupling, how about separating both?

yeah I was unsure but didnt' want to duplicate work. I will split it up, which will allow us to cleanup zombies less frequently

saschagrunert · 2021-08-30T08:27:07Z

internal/process/zombie_monitor.go

+		count, zombieChildren, err := ParseDefunctProcesses()
+		if err != nil {
+			log.Warnf(context.Background(), "Failed to get defunct process information: %v", err)
+		}
+		atomic.StoreUint32(&zm.zombieCount, uint32(count))


Should this run after the case <-time.After(time.Second * 5): below?

dropped now

saschagrunert · 2021-08-30T08:27:55Z

internal/process/zombie_monitor.go

+		case <-zm.closeChan:
+			// Since the process will soon shutdown, and its children will be reparented, no need to delay the shutdown to cleanup.
+			return
+		case <-time.After(time.Second * 5):


Is this too often? How about giving the whole cleanup more time, e.g. 15 minutes?

how about 5 minutes?

Sounds good

saschagrunert · 2021-08-30T08:28:28Z

internal/process/zombie_monitor.go

+		}
+		for _, child := range zombieChildren {
+			if _, err := syscall.Wait4(child, nil, syscall.WNOHANG, nil); err != nil {
+				log.Errorf(context.Background(), "Failed to reap child process %d: %v", child, err)


Let's use logrus if we do not have a context to add.

haircommander · 2021-09-08T20:57:44Z

comments addressed!

saschagrunert

LGTM

TomSweeneyRedHat · 2021-09-10T22:24:54Z

LGTM
but a few tests need some help

nee1esh · 2021-09-13T16:35:13Z

/retest-required

internal/process/defunct_processes.go

kolyshkin · 2021-09-13T16:43:08Z

internal/process/defunct_processes.go

-// a specific process filesystem.
-func DefunctProcessesForPath(path string) (defunctCount uint, retErr error) {
+// ParseDefunctProcessesForPath retrieves the number of zombie processes from
+// a specific process filesystem, as well as the number of defunct children of the current running process.


s/of the current running process/of a given parent/?

kolyshkin · 2021-09-13T16:46:22Z

internal/process/defunct_processes.go

+	ppidBegin := stateByte + 2
+	endOfPPid := strings.IndexRune(data[ppidBegin:], ' ')


Maybe ppidBegin / ppidEnd?

kolyshkin · 2021-09-13T16:46:35Z

internal/process/defunct_processes.go

 	// can have parentheses, so look for the last ')'.
-	i := strings.LastIndexByte(data, ')')
-	if i <= 2 || i >= len(data)-1 {
+	endOfComm := strings.LastIndexByte(data, ')')


kolyshkin · 2021-09-13T16:46:50Z

internal/process/defunct_processes.go

+
+	// the fourth field is PPid, and we can start looking after the space after State
+	ppidBegin := stateByte + 2
+	endOfPPid := strings.IndexRune(data[ppidBegin:], ' ')


Please use IndexByte here.

kolyshkin · 2021-09-13T16:49:14Z

internal/process/defunct_processes.go

 		return nil, errors.Errorf("invalid stat data (no comm): %q", data)
 	}

+	stateByte := endOfComm + 2


stateIdx?

kolyshkin · 2021-09-13T16:50:32Z

internal/process/zombie_monitor.go

+// 1: it is responsible for reporting the number of defunct PIDs currently on the node
+// 2: it is responsible for cleaning up zombies that are children of the currently running process


s/it is responsible for//

internal/process/defunct_processes.go

kolyshkin · 2021-09-13T17:05:37Z

@haircommander I think I missed the discussion on where those zombies are coming from, can you point me out to any details on that?

haircommander · 2021-09-13T17:22:07Z

@haircommander I think I missed the discussion on where those zombies are coming from, can you point me out to any details on that?

we've had a number of reports that there are conmon zombies that are leaking (https://bugzilla.redhat.com/show_bug.cgi?id=1952137 is one). We initially thought it was a bug in the exec implementation, but I am now thinking it was actually leaks that were closed in #5283.

Part of me is less convinced we need this anymore, though it may not be a bad idea to sweep for them periodically

Now that exec sync requests are run by conmon, there are more processes in the mix and more possibility for zombies this commit adds a zombie monitor to the defunct process metrics collection flow. it is a little clunky, but it would be weird to have two different /proc parsers for very similar uses Signed-off-by: Peter Hunt <[email protected]>

openshift-ci · 2021-09-15T17:49:55Z

@haircommander: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/integration_crun_cgroupv2	`4bdcabb`	link	false
ci/openshift-jenkins/integration_crun	`4bdcabb`	link	true
ci/openshift-jenkins/e2e_crun_cgroupv2	`4bdcabb`	link	false
ci/prow/e2e-gcp	`4bdcabb`	link	`/test e2e-gcp`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kolyshkin · 2021-09-16T16:26:03Z

internal/process/defunct_processes.go


 // Stat represents status information of a process from /proc/[pid]/stat.
 type Stat struct {
+	// Pid is the PID of the process


Nit: missing period at the end of sentence (feel free to ignore).

kolyshkin · 2021-09-16T16:26:12Z

internal/process/defunct_processes.go

 	// State is the state of the process.
 	State string
+
+	// PPid is the parent PID of the process


kolyshkin

Looks good overall, I'm just not sure we need a zombie reaper if there are no bugs in the code (and if they are, they need to be fixed).

openshift-ci · 2021-09-16T16:42:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, kolyshkin, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,kolyshkin,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2021-09-16T17:48:25Z

yeah I am leaning with you as well, especially becaus we've recently found a leak which is possibly the cause of some of the cases we've found. Closing for now

haircommander requested review from mrunalp and runcom as code owners August 27, 2021 16:28

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/feature Categorizes issue or PR as related to a new feature. labels Aug 27, 2021

openshift-ci bot requested a review from rhatdan August 27, 2021 16:28

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 27, 2021

haircommander force-pushed the subreaper branch from ec664c1 to dc74324 Compare August 27, 2021 16:29

haircommander force-pushed the subreaper branch from dc74324 to 5e6311f Compare August 27, 2021 17:13

saschagrunert reviewed Aug 30, 2021

View reviewed changes

haircommander force-pushed the subreaper branch from 5e6311f to 92d80c2 Compare September 8, 2021 20:57

saschagrunert approved these changes Sep 9, 2021

View reviewed changes

kolyshkin reviewed Sep 13, 2021

View reviewed changes

internal/process/defunct_processes.go Show resolved Hide resolved

kolyshkin reviewed Sep 13, 2021

View reviewed changes

internal/process/defunct_processes.go Show resolved Hide resolved

haircommander force-pushed the subreaper branch from 92d80c2 to 4bdcabb Compare September 13, 2021 19:30

kolyshkin reviewed Sep 16, 2021

View reviewed changes

kolyshkin approved these changes Sep 16, 2021

View reviewed changes

haircommander closed this Sep 16, 2021

		ppidBegin := stateByte + 2
		endOfPPid := strings.IndexRune(data[ppidBegin:], ' ')

		// 1: it is responsible for reporting the number of defunct PIDs currently on the node
		// 2: it is responsible for cleaning up zombies that are children of the currently running process

internal/process: add functionality to clean up zombie children #5260

internal/process: add functionality to clean up zombie children #5260

Uh oh!

Conversation

haircommander commented Aug 27, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

codecov bot commented Aug 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Sep 8, 2021

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

TomSweeneyRedHat commented Sep 10, 2021

Uh oh!

nee1esh commented Sep 13, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kolyshkin commented Sep 13, 2021

Uh oh!

haircommander commented Sep 13, 2021

Uh oh!

openshift-ci bot commented Sep 15, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Sep 16, 2021

Uh oh!

haircommander commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 27, 2021 •

edited

Loading

kolyshkin Sep 13, 2021 •

edited

Loading

haircommander commented Sep 16, 2021 •

edited

Loading