[1.18] oci: repeat $runtime state calls if appropriate #3866

haircommander · 2020-06-10T19:17:30Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

we have seen cases where $runtime state calls fail supriously, but succeed later
this is not great, though, we shouldn't incorrectly label pods if this happens.

We now retry state calls up to three times if we determine the container is still running (by calling kill on its pid)

also carry #3853

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

logging "Exec'd..." for every exec sync is too much, especially on Info if we fail to decode state, we should log why the `$runtime state` command failed Signed-off-by: Peter Hunt <[email protected]>

openshift-ci-robot · 2020-06-10T19:17:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-06-10T19:17:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2020-06-10T19:23:36Z

Codecov Report

Merging #3866 into release-1.18 will decrease coverage by 0.03%.
The diff coverage is 22.58%.

@@               Coverage Diff                @@
##           release-1.18    #3866      +/-   ##
================================================
- Coverage         40.47%   40.43%   -0.04%     
================================================
  Files               105      105              
  Lines              8636     8660      +24     
================================================
+ Hits               3495     3502       +7     
- Misses             4829     4845      +16     
- Partials            312      313       +1

mrunalp · 2020-06-10T19:29:27Z

internal/oci/container.go

+		return false
+	}
+
+	out, err := exec.Command("kill", "-0", strconv.Itoa(pid)).CombinedOutput()


Let's make a syscall instead of spawning here.

haircommander · 2020-06-10T19:43:31Z

/retest

haircommander · 2020-06-10T19:57:54Z

/retest

haircommander · 2020-06-10T20:16:33Z


# time="2020-06-10 19:57:30.837507507Z" level=debug msg="Request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},},}" file="go-grpc-middleware/chain.go:25" id=eab6ca6e-7c6b-458b-b269-24b0e6513979 name=/runtime.v1alpha2.RuntimeService/ListContainers

/retest

tedyu · 2020-06-10T20:33:56Z

internal/oci/runtime_oci.go

+		if err != nil {
+			logrus.Errorf("%v: attempt %d", err, i)
+		}
+		if out != nil {


Should this be in the else block of the above if statement ?

fixed the order of these

haircommander · 2020-06-10T20:40:09Z

[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount
flaky test perhaps?
/test e2e_cgroupv2

we have seen cases where $runtime state calls fail supriously, but succeed later this is not great, though, we shouldn't incorrectly label pods if this happens. We now retry state calls up to three times if we determine the container is still running (by calling kill on its pid) Signed-off-by: Peter Hunt <[email protected]>

tedyu · 2020-06-10T21:05:20Z

internal/oci/runtime_oci.go

+	// went away we do not error out stopping kubernetes to recover.
+	// We always populate the fields below so kube can restart/reschedule
+	// containers failing.
+	if out == nil {


Should err be checked here ? The loop above may exit after the 3rd attempt and by that time, err may still be non-nil.

I guess the assumption is that if out is nil, then err is populated. we only need to check one, so which doesn't really matter.

haircommander · 2020-06-10T21:17:13Z

/retest

haircommander · 2020-06-11T13:56:36Z

/retest

mrunalp · 2020-06-11T14:18:24Z

internal/oci/container.go

+// it is used to check a container state when we don't want (or don't trust) a `$runtime state` call
+func (c *Container) IsRunning() bool {
+	pid := c.state.Pid
+	process, err := os.FindProcess(pid)


Is this necessary? We could directly make the syscall since we already know the pid.

I suppose not, though I personally would rather use the standard lib instead of directly using syscall. I am worried about having to handle things that the lib takes care of (this isn't a great example, but for go 1.14, longer syscalls need eintr handling https://golang.org/doc/go1.14#runtime). My inclination is to leave as is

os.FindProcess(pid) doesn't add any extra cost on unix as it just populates a struct with the specified PID.

https://github.com/golang/go/blob/cb11c981df7b4dc40550ab71cc097c25d24d7a71/src/os/exec_unix.go#L97 👍

haircommander · 2020-06-11T14:44:57Z

/test e2e-aws

giuseppe · 2020-06-11T15:10:50Z

internal/oci/runtime_oci.go

+		if err == nil {
+			break
+		}
+		logrus.Errorf("%v: attempt %d", err, i)


could we add more context to this error?

the context comes from the called function:
failed to update container state for %s: %v", c.id, err
I couldn't think of anything else to add out here that wouldn't be redundant. Is there anything you'd like to see added?

giuseppe · 2020-06-11T15:13:16Z

internal/oci/container.go

+// it is used to check a container state when we don't want (or don't trust) a `$runtime state` call
+func (c *Container) IsRunning() bool {
+	pid := c.state.Pid
+	process, err := os.FindProcess(pid)


os.FindProcess(pid) doesn't add any extra cost on unix as it just populates a struct with the specified PID.

haircommander · 2020-06-11T17:07:11Z

/test e2e-aws

haircommander · 2020-06-11T18:32:38Z

/hold
entirely possible we can get away without this

haircommander · 2020-06-17T13:26:29Z

we have identified the underlying issue in runc. it's best not to hide flakes like this pr does.

logs: fix some problems

0b9d252

logging "Exec'd..." for every exec sync is too much, especially on Info if we fail to decode state, we should log why the `$runtime state` command failed Signed-off-by: Peter Hunt <[email protected]>

haircommander requested review from mrunalp and runcom as code owners June 10, 2020 19:17

openshift-ci-robot added the dco-signoff: yes Indicates the PR's author has DCO signed all their commits. label Jun 10, 2020

openshift-ci-robot requested review from giuseppe, sameo, umohnani8 and vrothberg June 10, 2020 19:17

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2020

haircommander mentioned this pull request Jun 10, 2020

[1.18] logs: fix some problems #3853

Closed

haircommander force-pushed the log-updates-1.18-2 branch from 8dc2348 to e5d62c6 Compare June 10, 2020 19:22

haircommander mentioned this pull request Jun 10, 2020

oci: repeat $runtime state calls if appropriate #3865

Closed

mrunalp reviewed Jun 10, 2020

View reviewed changes

haircommander force-pushed the log-updates-1.18-2 branch from e5d62c6 to c8603ed Compare June 10, 2020 19:42

tedyu reviewed Jun 10, 2020

View reviewed changes

haircommander force-pushed the log-updates-1.18-2 branch from c8603ed to e85e2cc Compare June 10, 2020 20:57

tedyu reviewed Jun 10, 2020

View reviewed changes

haircommander mentioned this pull request Jun 11, 2020

reduce flakiness of UpdateContainerStatus #3867

Closed

mrunalp reviewed Jun 11, 2020

View reviewed changes

giuseppe reviewed Jun 11, 2020

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2020

haircommander closed this Jun 17, 2020

haircommander deleted the log-updates-1.18-2 branch September 27, 2021 16:54

[1.18] oci: repeat $runtime state calls if appropriate #3866

[1.18] oci: repeat $runtime state calls if appropriate #3866

Conversation

haircommander commented Jun 10, 2020

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Jun 10, 2020

Uh oh!

openshift-ci-robot commented Jun 10, 2020

Uh oh!

codecov bot commented Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 10, 2020

Uh oh!

haircommander commented Jun 10, 2020

Uh oh!

haircommander commented Jun 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 10, 2020

Uh oh!

haircommander commented Jun 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jun 11, 2020

Uh oh!

haircommander commented Jun 11, 2020

Uh oh!

haircommander commented Jun 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Jun 10, 2020 •

edited

Loading