utils/RunUnderSystemdScope: fix wrt channel deadlock #5922

kolyshkin · 2022-05-31T23:38:08Z

As seen in [1], sometimes coreos/go-systemd/dbus package deadlocks: the
jobCompete is stuck trying to send job result string to the channel
while holding the jobListener lock, while startJob (called by
StartTransientUnit) waits for the same lock.

Alas, it is not clear why the channel is not being read, nor was I able
to reproduce it locally.

Make the job result channel buffered, so jobJistener won't block on
channel send and thus StartTransientUnit won't be stuck either.

While at it,

move the error wrapping out of mgr.RetryOnDisconnect function,
and use fmt.Errorf with %w instead of obsoleted errors.Wrap;
improve error messages, printing the systemd unit name (so we can
check it in systemd log);
do check the job result string -- in case it is not "done",
return an error back to the caller, which should help avoid other
issues down the line.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2082344

/kind bug

Fix a rare deadlock while communicating to systemd (RHBZ 2082344)

This is a forward-port of #5914 to main branch.

saschagrunert

LGTM

haircommander · 2022-06-01T13:29:44Z

/retest

nee1esh · 2022-06-01T15:58:18Z

/retest-required

kolyshkin · 2022-06-01T17:50:19Z

Test failures seem to be unrelated.

haircommander · 2022-06-01T17:52:45Z

e2e_fedora and e2e_crun* are unrelated (being investigated in #5924). however, agnostic and gcp may not be

nee1esh · 2022-06-02T02:42:39Z

/retest-required

As seen in [1], sometimes coreos/go-systemd/dbus package deadlocks: the jobCompete is stuck trying to send job result string to the channel while holding the jobListener lock, while startJob (called by StartTransientUnit) waits for the same lock. Alas, it is not clear why the channel is not being read, nor was I able to reproduce it locally. Make the job result channel buffered, so jobJistener won't block on channel send and thus StartTransientUnit won't be stuck either. While at it, - move the error wrapping out of mgr.RetryOnDisconnect function, and use fmt.Errorf with %w instead of obsoleted errors.Wrap; - improve error messages, printing the systemd unit name (so we can check it in systemd log); - do check the job result string -- in case it is not "done", return an error back to the caller, which should help avoid other issues down the line. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2082344 Signed-off-by: Kir Kolyshkin <[email protected]>

kolyshkin · 2022-06-02T19:10:49Z

rebased

kolyshkin · 2022-06-02T20:59:45Z

Failures in e2e-gcp and e2e-agnostic are the same as in #5506 which makes me think they are unrelated to this PR (or #5506).

Failures in e2e-cgroupv2 and e2e-fedora:

Kubernetes e2e suite: [sig-network] Services should be rejected for evicted pods (no endpoints exist) expand_less
Kubernetes e2e suite: [sig-node] Pods Extended Pod Container lifecycle evicted pods should be terminal expand_more

are the same as in e.g. and #5917, so also seem unrelated.

kolyshkin · 2022-06-03T01:12:42Z

/retest

nee1esh · 2022-06-08T01:06:09Z

/retest

haircommander · 2022-06-08T18:38:22Z

/override ci/prow/e2e-gcp
/override ci/prow/e2e-agnostic
/lgtm

openshift-ci · 2022-06-08T18:38:30Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/e2e-agnostic, ci/prow/e2e-gcp

Details

In response to this:

/override ci/prow/e2e-gcp
/override ci/prow/e2e-agnostic
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2022-06-08T18:38:37Z

/cherry-pick release-1.24

openshift-cherrypick-robot · 2022-06-08T18:38:39Z

@haircommander: once the present PR merges, I will cherry-pick it on top of release-1.24 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-1.24

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-06-08T18:46:57Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-08T18:59:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-08T19:12:57Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T06:35:11Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T06:47:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T06:59:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

saschagrunert · 2022-06-09T07:09:29Z

/override ci/prow/e2e-gcp

openshift-ci · 2022-06-09T07:09:34Z

@saschagrunert: Overrode contexts on behalf of saschagrunert: ci/prow/e2e-gcp

Details

In response to this:

/override ci/prow/e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-06-09T07:09:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kolyshkin, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kolyshkin,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-06-09T07:11:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T07:23:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T07:35:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T07:47:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T07:59:11Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T08:11:12Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T08:23:15Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T08:35:12Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T08:47:11Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T08:59:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T09:11:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

saschagrunert · 2022-06-09T09:20:46Z

/override ci/prow/e2e-agnostic

openshift-ci · 2022-06-09T09:20:51Z

@saschagrunert: Overrode contexts on behalf of saschagrunert: ci/prow/e2e-agnostic

Details

In response to this:

/override ci/prow/e2e-agnostic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-06-09T09:23:12Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T09:35:11Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T09:47:12Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T09:59:14Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T10:12:03Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-06-09T10:23:56Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-cherrypick-robot · 2022-06-09T10:35:12Z

@haircommander: new pull request created: #5954

Details

In response to this:

/cherry-pick release-1.24

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kolyshkin · 2022-08-03T20:55:53Z

utils/utils.go

 	case <-ch:
 		close(ch)


Oops, this should not be there.

Opened #6124 to address it.

kolyshkin requested review from mrunalp and runcom as code owners May 31, 2022 23:38

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/bug Categorizes issue or PR as related to a bug. labels May 31, 2022

openshift-ci bot requested review from klihub and wgahnagl May 31, 2022 23:38

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 31, 2022

kolyshkin mentioned this pull request May 31, 2022

[1.21] utils/RunUnderSystemdScope: fix wrt channel deadlock #5914

Merged

saschagrunert approved these changes Jun 1, 2022

View reviewed changes

kolyshkin force-pushed the dbus-deadlock branch from 7599845 to 1e277b8 Compare June 2, 2022 19:10

openshift-ci bot assigned haircommander Jun 8, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2022

saschagrunert approved these changes Jun 9, 2022

View reviewed changes

openshift-merge-robot merged commit 1e5e7b4 into cri-o:main Jun 9, 2022

openshift-cherrypick-robot mentioned this pull request Jun 9, 2022

[release-1.24] utils/RunUnderSystemdScope: fix wrt channel deadlock #5954

Merged

kolyshkin commented Aug 3, 2022

View reviewed changes

utils/RunUnderSystemdScope: fix wrt channel deadlock #5922

utils/RunUnderSystemdScope: fix wrt channel deadlock #5922

Uh oh!

Conversation

kolyshkin commented May 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 1, 2022

Uh oh!

nee1esh commented Jun 1, 2022

Uh oh!

kolyshkin commented Jun 1, 2022

Uh oh!

haircommander commented Jun 1, 2022

Uh oh!

nee1esh commented Jun 2, 2022

Uh oh!

kolyshkin commented Jun 2, 2022

Uh oh!

kolyshkin commented Jun 2, 2022

Uh oh!

kolyshkin commented Jun 3, 2022

Uh oh!

nee1esh commented Jun 8, 2022

Uh oh!

haircommander commented Jun 8, 2022

Uh oh!

openshift-ci bot commented Jun 8, 2022

Uh oh!

haircommander commented Jun 8, 2022

Uh oh!

openshift-cherrypick-robot commented Jun 8, 2022

Uh oh!

openshift-bot commented Jun 8, 2022

Uh oh!

openshift-bot commented Jun 8, 2022

Uh oh!

openshift-bot commented Jun 8, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

saschagrunert commented Jun 9, 2022

Uh oh!

openshift-ci bot commented Jun 9, 2022

Uh oh!

openshift-ci bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

openshift-bot commented Jun 9, 2022

Uh oh!

saschagrunert commented Jun 9, 2022

Uh oh!

openshift-ci bot commented Jun 9, 2022

Uh oh!

kolyshkin commented May 31, 2022 •

edited

Loading