server: refactor handling of cleanup funcs #4241

haircommander · 2020-10-02T18:16:36Z

What type of PR is this?

/kind cleanup

/kind dependency-change
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

it doesn't make very much sense to have so many deferred funcs queued, and check retErr each time
instead, we can check retErr once, and loop through a slice of cleanupFuncs

this also carries #4240 to avoid a merge conflict

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

none

haircommander · 2020-10-02T18:19:28Z

/retest

haircommander · 2020-10-02T20:36:03Z

/retest

codecov · 2020-10-02T21:01:06Z

Codecov Report

Merging #4241 (8dd12dc) into master (61c8b37) will decrease coverage by 0.04%.
The diff coverage is 26.38%.

@@            Coverage Diff             @@
##           master    #4241      +/-   ##
==========================================
- Coverage   39.26%   39.22%   -0.05%     
==========================================
  Files         112      112              
  Lines        8839     8819      -20     
==========================================
- Hits         3471     3459      -12     
+ Misses       4982     4973       -9     
- Partials      386      387       +1

haircommander · 2020-10-03T17:49:50Z

/retest

haircommander · 2020-10-05T20:13:36Z

/retest

haircommander · 2020-10-05T20:13:55Z

pulled in #4198 to make sure that deferred func is included

kolyshkin · 2020-10-05T20:26:17Z

server/container_create_linux.go

+			err2 := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID)
+			if err2 != nil {


Suggested change

err2 := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID)

if err2 != nil {

err := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID); if err != nil {

server/container_create.go

kolyshkin · 2020-10-05T20:28:23Z

server/container_create.go

+	defer func() {
+		if retErr != nil {
+			log.Infof(ctx, "createCtr: removing container ID %s from runtime", ctr.ID())
+			if err2 := s.Runtime().DeleteContainer(newContainer); err2 != nil {


doesn't have to be err2.

kolyshkin · 2020-10-05T20:28:53Z

server/container_create_linux.go

 	}
+	defer func() {
+		if retErr != nil {
+			log.Infof(ctx, "createCtrLinux: deleting container %s from storage", containerInfo.ID)


the function is named createSandboxContainer

kolyshkin · 2020-10-05T20:33:52Z

I don't like the last commit much, it seems it complicates things (and decreases visibility) more than improves anything. defer is cheap in Go 1.15+, != nil check is cheap, too.

haircommander · 2020-10-05T20:38:41Z

I don't like the last commit much, it seems it complicates things (and decreases visibility) more than improves anything. defer is cheap in Go 1.15+, != nil check is cheap, too.

Yeah, it's cheap, but doing it so many times is not needed. The main purpose of this PR is preparing for a larger refactor that requires we not cleanup when the error is not a context timeout. That'll add two extra checks: err != context.Canceled && err != context.DeadlineExceeded along with the non-nil check, which I think better justifies the PR.

openshift-ci-robot · 2020-10-12T20:36:34Z

@haircommander: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/integration_crun_cgroupv2	`ea01c41`	link	`/test integration_cgroupv2`

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2020-10-14T19:03:36Z

/retest

fidencio

LGTM, there's a nitpick comment, but feel free to ignore it.

fidencio · 2020-11-12T21:47:57Z

internal/lib/sandbox/sandbox.go

+	// cleaning up a failed sandbox creation.
+	// We don't need to create the file, as there will be no
+	// sandbox to restore
+	if !s.created {


Isn't this the same case as the one pointed out by @kolyshkin here: #4258 (review) ?

I'll rebase on #4241 once it merges :)

haircommander · 2020-11-20T14:26:25Z

this is ready @fidencio @umohnani8 @mrunalp @kolyshkin @giuseppe @saschagrunert

haircommander · 2020-11-20T14:32:21Z

/retest

TomSweeneyRedHat · 2020-11-20T14:36:37Z

server/container_create.go

+		log.Infof(ctx, "createCtr: deleting container %s from storage", ctr.ID())
+		err2 := s.StorageRuntimeServer().DeleteContainer(ctr.ID())
+		if err2 != nil {
+			log.Warnf(ctx, "Failed to cleanup container directory: %v", err2)


soft suggest "directory" -> "storage"

TomSweeneyRedHat · 2020-11-20T14:39:23Z

server/sandbox_run_linux.go

+		cleanupFuncs = append(cleanupFuncs, func() {
+			log.Infof(ctx, "runSandbox: unmounting shmPath for sandbox %s", sbox.ID())
+			if err2 := unix.Unmount(shmPath, unix.MNT_DETACH); err2 != nil {
+				log.Warnf(ctx, "failed to unmount shm for pod: %v", err2)


you say "for pod" here, but "for sandbox" at line 543? I'd be consistent.

server/sandbox_run_linux.go

TomSweeneyRedHat · 2020-11-20T14:41:20Z

server/sandbox_run_linux.go

+	cleanupFuncs = append(cleanupFuncs, func() {
+		log.Infof(ctx, "runSandbox: deleting pod ID %s from idIndex", sbox.ID())
+		if err := s.PodIDIndex().Delete(sbox.ID()); err != nil {
+			log.Warnf(ctx, "couldn't delete pod id %s from idIndex", sbox.ID())


Suggested change

log.Warnf(ctx, "couldn't delete pod id %s from idIndex", sbox.ID())

log.Warnf(ctx, "couldn't delete pod ID %s from idIndex", sbox.ID())

TomSweeneyRedHat · 2020-11-20T14:44:01Z

server/sandbox_run_linux.go

-					log.Errorf(ctx, "error stopping network on cleanup: %v", err2)
-				}
+		cleanupFuncs = append(cleanupFuncs, func() {
+			log.Infof(ctx, "runSandbox: in not manageNSLifecycle, stopping network for sandbox %s", sb.ID())


Perhaps?

Suggested change

log.Infof(ctx, "runSandbox: in not manageNSLifecycle, stopping network for sandbox %s", sb.ID())

log.Infof(ctx, "runSandbox: is not in manageNSLifecycle, stopping network for sandbox %s", sb.ID())

TomSweeneyRedHat · 2020-11-20T14:45:11Z

done picking. Just a few suggestions for consideration, only message near the end that was badly formed and should really be reworked.

Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2020-11-20T15:22:21Z

nits addressed!

haircommander · 2020-11-20T16:13:38Z

/retest

haircommander · 2020-11-20T18:19:27Z

/retest

kolyshkin · 2020-11-20T21:09:54Z

server/container_create.go

 		return nil, errors.Wrap(err, "setting container name and ID")
 	}

+	cleanupFuncs := make([]func(), 0)


I think , 0 is redundant

you need to specify a length when initializing a slice

Right, I always forget about this.

cleanupFuncs := []func(){}

maybe?

is there any reason that is better than using make (which feels more idiomatic to me)

kolyshkin · 2020-11-20T21:37:03Z

Side note: there is a way that looks nice, does not require defers, named return values, or repeated error checks. It only requires labels and goto.

Something like this (pseudocode):

var retErr error

err: = do_a(); if err != nil {
   retErr = errors.Wrap(err, "failed to do a")
   goto undo_a
}

if err := do_b(); err != nil {
   retErr = errors.Wrap(err, "failed to do b")
   goto undo_b
}

if err := do_c(); err != nil {
   retErr = errors.Wrap(err, "failed to do c")
   goto undo_c
}

return nil

undo_c:
   undo_c()
undo_b:
   undo_b()
undo_a:
   undo_a()
log(retErr)
return retErr

Surely, wrapping and logging are optional. Basically it is like lots of nested ifs, but without the ugliness of too much nesting.

haircommander · 2020-11-23T14:36:26Z

/retest

haircommander · 2020-11-23T20:55:53Z

Side note: there is a way that looks nice, does not require defers, named return values, or repeated error checks. It only requires labels and goto.
....
Surely, wrapping and logging are optional. Basically it is like lots of nested ifs, but without the ugliness of too much nesting.

I'm not sure I want to go down this route, I'm not crazy about gotos and I don't think it really cleans up the code all that much. Right now, this change is pretty small and will be useful later, are there any blocking comments left

@kolyshkin @giuseppe @mrunalp @umohnani8 @saschagrunert @TomSweeneyRedHat

saschagrunert

LGTM

openshift-ci-robot · 2020-11-24T08:32:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2020-11-30T15:24:40Z

PTAL @umohnani8 @mrunalp @kolyshkin @saschagrunert

mrunalp · 2020-11-30T15:40:16Z

/lgtm

haircommander · 2020-12-02T17:15:05Z

/retest

openshift-merge-robot · 2020-12-02T20:15:30Z

@haircommander: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-agnostic	`8dd12dc`	link	`/test e2e-agnostic`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2020-12-02T20:16:17Z

e2e-gcp passed, force merging instead of wasting more cycles on flakes

haircommander requested review from mrunalp and runcom as code owners October 2, 2020 18:16

openshift-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Oct 2, 2020

openshift-ci-robot requested a review from sameo October 2, 2020 18:16

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 2, 2020

haircommander force-pushed the cleanup-func-refactor branch from 289d5e3 to 6da311a Compare October 2, 2020 19:56

haircommander force-pushed the cleanup-func-refactor branch from 6da311a to 33ad337 Compare October 2, 2020 20:47

haircommander force-pushed the cleanup-func-refactor branch from 33ad337 to 7f2b2c3 Compare October 5, 2020 20:10

kolyshkin reviewed Oct 5, 2020

View reviewed changes

server/container_create.go Show resolved Hide resolved

kolyshkin reviewed Oct 5, 2020

View reviewed changes

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 11, 2020

haircommander force-pushed the cleanup-func-refactor branch from 7f2b2c3 to ea01c41 Compare October 12, 2020 20:14

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 12, 2020

haircommander mentioned this pull request Oct 14, 2020

WIP: improve timeout handling #4266

Closed

haircommander force-pushed the cleanup-func-refactor branch from ea01c41 to 6e724c4 Compare October 14, 2020 18:02

fidencio reviewed Nov 12, 2020

View reviewed changes

haircommander force-pushed the cleanup-func-refactor branch from 6e724c4 to 7244e40 Compare November 20, 2020 14:25

TomSweeneyRedHat reviewed Nov 20, 2020

View reviewed changes

server/sandbox_run_linux.go Show resolved Hide resolved

TomSweeneyRedHat reviewed Nov 20, 2020

View reviewed changes

fix cleanup func wording

8dd12dc

Signed-off-by: Peter Hunt <[email protected]>

kolyshkin reviewed Nov 20, 2020

View reviewed changes

saschagrunert approved these changes Nov 24, 2020

View reviewed changes

openshift-ci-robot assigned mrunalp Nov 30, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 30, 2020

haircommander added this to the 1.20 milestone Nov 30, 2020

haircommander mentioned this pull request Dec 2, 2020

Improve timeout handling #4394

Merged

haircommander merged commit 4dff9dd into cri-o:master Dec 2, 2020

haircommander mentioned this pull request Dec 17, 2020

[1.19] improve timeout handling #4444

Merged

		err2 := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID)
		if err2 != nil {

	err2 := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID)
	if err2 != nil {
	err := s.StorageRuntimeServer().DeleteContainer(containerInfo.ID); if err != nil {

	log.Warnf(ctx, "couldn't delete pod id %s from idIndex", sbox.ID())
	log.Warnf(ctx, "couldn't delete pod ID %s from idIndex", sbox.ID())

	log.Infof(ctx, "runSandbox: in not manageNSLifecycle, stopping network for sandbox %s", sb.ID())
	log.Infof(ctx, "runSandbox: is not in manageNSLifecycle, stopping network for sandbox %s", sb.ID())

server: refactor handling of cleanup funcs #4241

server: refactor handling of cleanup funcs #4241

Uh oh!

Conversation

haircommander commented Oct 2, 2020

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

haircommander commented Oct 2, 2020

Uh oh!

haircommander commented Oct 2, 2020

Uh oh!

codecov bot commented Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haircommander commented Oct 3, 2020

Uh oh!

haircommander commented Oct 5, 2020

Uh oh!

haircommander commented Oct 5, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented Oct 5, 2020

Uh oh!

haircommander commented Oct 5, 2020

Uh oh!

openshift-ci-robot commented Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Oct 14, 2020

Uh oh!

fidencio left a comment

Choose a reason for hiding this comment

Uh oh!

fidencio Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Nov 20, 2020

Uh oh!

haircommander commented Nov 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomSweeneyRedHat commented Nov 20, 2020

Uh oh!

haircommander commented Nov 20, 2020

Uh oh!

haircommander commented Nov 20, 2020

Uh oh!

haircommander commented Nov 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

codecov bot commented Oct 2, 2020 •

edited

Loading

openshift-ci-robot commented Oct 12, 2020 •

edited

Loading

fidencio Nov 12, 2020 •

edited

Loading

kolyshkin commented Nov 20, 2020 •

edited

Loading