Use timeout for conmon cgroup move #5535

haircommander · 2022-01-10T15:32:19Z

We're now using a timeout which is able to unblock gaining
the sandbox mutex in a container creation vs sandbox stop race.

Signed-off-by: Sascha Grunert [email protected]
Signed-off-by: Peter Hunt [email protected]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

none

We're now using a timeout which is able to unblock gaining the sandbox mutex in a container creation vs sandbox stop race. Signed-off-by: Sascha Grunert <[email protected]> Signed-off-by: Peter Hunt <[email protected]>

codecov · 2022-01-10T15:39:05Z

Codecov Report

Merging #5535 (fd2e2ae) into main (9b7f5ae) will decrease coverage by 0.01%.
The diff coverage is 28.57%.

@@            Coverage Diff             @@
##             main    #5535      +/-   ##
==========================================
- Coverage   43.11%   43.10%   -0.02%     
==========================================
  Files         121      121              
  Lines       12148    12150       +2     
==========================================
- Hits         5238     5237       -1     
- Misses       6407     6410       +3     
  Partials      503      503

TomSweeneyRedHat · 2022-01-10T19:27:39Z

Change LGTM, but 6 minutes seems a bit long to wait.
Couple of tests still aren't hip.

mrunalp

/lgtm

openshift-ci · 2022-01-10T21:25:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-01-10T21:39:05Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-01-10T22:44:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-01-10T22:57:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-01-10T23:10:09Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-01-10T23:15:49Z

@haircommander: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/openshift-jenkins/integration_crun_cgroupv2	`fd2e2ae`	link	false	`/test integration_cgroupv2`
ci/openshift-jenkins/e2e_crun_cgroupv2	`fd2e2ae`	link	false	`/test e2e_cgroupv2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2022-01-10T23:23:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-01-11T00:15:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

haircommander · 2022-01-12T21:34:15Z

/cherry-pick release-1.23

openshift-cherrypick-robot · 2022-01-12T21:35:04Z

@haircommander: new pull request created: #5540

Details

In response to this:

/cherry-pick release-1.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2022-01-17T18:02:00Z

/cherry-pick release-1.22

openshift-cherrypick-robot · 2022-01-17T18:02:49Z

@haircommander: new pull request created: #5545

Details

In response to this:

/cherry-pick release-1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

saschagrunert · 2022-01-18T09:48:08Z

utils/utils.go

 	if err := mgr.RetryOnDisconnect(func(c *systemdDbus.Conn) error {
 		_, err = c.StartTransientUnitContext(ctx, unitName, "replace", properties, ch)
-		return err
+		return errors.Wrap(err, "start transient unit")
 	}); err != nil {
 		return err
 	}

 	// Block until job is started
-	<-ch
-	close(ch)
+	select {
+	case <-ch:
+		close(ch)
+	case <-time.After(time.Minute * 6):
+		// This case is a work around to catch situations where the dbus library sends the
+		// request but it unexpectedly disappears. We set the timeout very high to make sure
+		// we wait as long as possible to catch situations where dbus is overwhelmed.
+		// We also don't use the native context cancelling behavior of the dbus library,
+		// because experience has shown that it does not help.
+		// TODO: Find cause of the request being dropped in the dbus library and fix it.
+		return errors.Errorf("timed out moving conmon with pid %d to cgroup", pid)
+	}


Two ideas coming into my mind:

What if we reduce the timeout to 1 minute and retry multiple times with an exponential backoff?

What if we check for the error "Message recipient disconnected from message bus without replying" and restart dbus in that case?

how would we be able to get the error in this case?

haircommander · 2022-02-03T20:21:24Z

/cherry-pick release-1.20

openshift-cherrypick-robot · 2022-02-03T20:22:01Z

@haircommander: #5535 failed to apply on top of branch "release-1.20":

Applying: Use timeout for conmon cgroup move
Using index info to reconstruct a base tree...
M	utils/utils.go
Falling back to patching base and 3-way merge...
Auto-merging utils/utils.go
CONFLICT (content): Merge conflict in utils/utils.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Use timeout for conmon cgroup move
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Use timeout for conmon cgroup move

fd2e2ae

We're now using a timeout which is able to unblock gaining the sandbox mutex in a container creation vs sandbox stop race. Signed-off-by: Sascha Grunert <[email protected]> Signed-off-by: Peter Hunt <[email protected]>

haircommander requested review from mrunalp and runcom as code owners January 10, 2022 15:32

openshift-ci bot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jan 10, 2022

openshift-ci bot requested review from sboeuf and vrothberg January 10, 2022 15:32

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 10, 2022

mrunalp approved these changes Jan 10, 2022

View reviewed changes

openshift-ci bot assigned mrunalp Jan 10, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 10, 2022

openshift-merge-robot merged commit a8a5653 into cri-o:main Jan 11, 2022

openshift-cherrypick-robot mentioned this pull request Jan 12, 2022

[release-1.23] Use timeout for conmon cgroup move #5540

Merged

openshift-cherrypick-robot mentioned this pull request Jan 17, 2022

[release-1.22] Use timeout for conmon cgroup move #5545

Merged

saschagrunert reviewed Jan 18, 2022

View reviewed changes

This was referenced Feb 3, 2022

[1.20] various deadlock backports #5597

Merged

[1.19] various deadlock fixes #5617

Merged

Use timeout for conmon cgroup move #5535

Use timeout for conmon cgroup move #5535

Uh oh!

Conversation

haircommander commented Jan 10, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

codecov bot commented Jan 10, 2022

Codecov Report

Uh oh!

TomSweeneyRedHat commented Jan 10, 2022

Uh oh!

mrunalp left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jan 10, 2022

Uh oh!

openshift-bot commented Jan 10, 2022

Uh oh!

openshift-bot commented Jan 10, 2022

Uh oh!

openshift-bot commented Jan 10, 2022

Uh oh!

openshift-bot commented Jan 10, 2022

Uh oh!

openshift-ci bot commented Jan 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Jan 10, 2022

Uh oh!

openshift-bot commented Jan 11, 2022

Uh oh!

haircommander commented Jan 12, 2022

Uh oh!

openshift-cherrypick-robot commented Jan 12, 2022

Uh oh!

haircommander commented Jan 17, 2022

Uh oh!

openshift-cherrypick-robot commented Jan 17, 2022

Uh oh!

saschagrunert Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

haircommander Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

haircommander commented Feb 3, 2022

Uh oh!

openshift-cherrypick-robot commented Feb 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

openshift-ci bot commented Jan 10, 2022 •

edited

Loading