Stop probing a pod during graceful shutdown #98571
Conversation
|
/cc @SergeyKanzhelev |
|
/priority important-soon |
|
@ehashman: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/triage accepted |
pkg/kubelet/prober/worker.go
Outdated
There was a problem hiding this comment.
We don't need to do any container checking after we see the pod deletion timestamp is set so, in this case, should we not run this logic before we do any container checking? (i.e. move this addition above L191)
There was a problem hiding this comment.
well I need to have a valid w.containerID to set the probe results, don't I?
There was a problem hiding this comment.
Ahh yes sure. so maybe before the onHold check?
There was a problem hiding this comment.
(I guess it doesn't really matter as much, this is nonblocking)
There was a problem hiding this comment.
Yeah, in fact we probably have more often c.State.Running == nil than w.pod.ObjectMeta.DeletionTimestamp != nil.
pkg/kubelet/prober/worker.go
Outdated
There was a problem hiding this comment.
I don't know if we actually need to set onHold. It's only used here:
kubernetes/pkg/kubelet/prober/worker.go
Line 209 in 64a6ee4
But we're returning false so the worker for this probe will never run again anyways.
There was a problem hiding this comment.
True, I forgot the container won't probably restart if we asked for the pod to be deleted... removing it.
|
/retest |
My question stays the same as for the initial PR. What is the safe defaults for the probes that work universally? Would one want to mark readiness probe as |
Yeah, probably it is safer to let the application owner decide what to do with the |
|
/assign @yujuhong |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: matthyx, mrunalp The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test pull-kubernetes-e2e-kind |
…71-upstream-release-1.18 Automated cherry pick of #98571: Stop probing a pod during graceful shutdown
…71-upstream-release-1.19 Automated cherry pick of #98571: Stop probing a pod during graceful shutdown
…71-upstream-release-1.20 Automated cherry pick of #98571: Stop probing a pod during graceful shutdown
| // Stop probing for startup once container has started. | ||
| if w.probeType == startup { | ||
| return true | ||
| return false |
There was a problem hiding this comment.
startupProbe behaviour changed
After the container restarts, startupProbe will no longer be executed
#101093
What type of PR is this?
/kind bug
What this PR does / why we need it:
If a pod takes longer to delete than it's its health/readiness probe allow a user can get errors that the pods check is erroring even though the pod is being deleted. This can update the pods status field to be inaccurate.
Which issue(s) this PR fixes:
Fixes #52817
Special notes for your reviewer:
This PR reopens the work initiated by @ashleyschuett in #92385 which was abandoned partly because of my comments there. We still need to fix this situation.
I propose basically the same approach which is altering kubelet's worker working when
DeletionTimestampis set (indicating graceful shutdown initiated) to allow a quiet shutdown by setting safe probe results (depending on the type) before disabling probing.Does this PR introduce a user-facing change?:
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: