-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Fix events generated by Evented PLEG #6633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Hi @sairameshv. Thanks for your PR. I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6633 +/- ##
=======================================
Coverage 49.71% 49.71%
=======================================
Files 127 127
Lines 14982 14982
=======================================
Hits 7449 7449
Misses 6646 6646
Partials 887 887 |
|
/ok-to-test |
|
/retest |
|
/cc @harche |
|
Verified this change by running the node e2e tests and no test has failed as follows: Also verified the behavior by running a simple pod config and observed that all the events are generated as expected. |
|
/lgtm |
|
/retest |
4780f78 to
e1bd050
Compare
|
/lgtm |
|
|
||
| log.Infof(ctx, "Stopped pod sandbox: %s", sb.ID()) | ||
| sb.SetStopped(ctx, true) | ||
| s.generateCRIEvent(ctx, sb.InfraContainer(), types.ContainerEventType_CONTAINER_STOPPED_EVENT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the kubelet expect a stop for pod infra container or does it need a separate pod stop event?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CRI events are generated per container, which also includes infra container. If for some reason infra container runs to completion the ContainerEventType_CONTAINER_STOPPED_EVENT will be generated by crio in handleExit() function. This event will be received and forwarded to the syncLoop in Kubelet in the form of PodLifeCycle event.
This PR is fixing the issue where we were incorrectly generating ContainerEventType_CONTAINER_STOPPED_EVENT while intentionally stopping the infra container rather than generating it when that infra container runs to completion (for which the relevant code in handleExit() already existed as mentioned above)
In retrospect, I feel we should have named ContainerEventType_CONTAINER_STOPPED_EVENT as ContainerEventType_CONTAINER_EXITED_EVENT because it is generating some confusion with action that is stopping the container (e.g. #6531 (comment))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we may need to have a call to this when the infra container is dropped (can tell with podInfraContainer.Spoofed() being true). This is because a spoofed container (dropped infra) doesn't have a process that can exit, so it's possible the kubelet will be waiting for a CONTAINER_STOPPED_EVENT for that infra container forever
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@haircommander do you have a reproducer for this scenario?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you create a pod that doesn't share a pod level pid namespace (shareProcessNamespace: false) then the infra container will be spoofed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @haircommander , I added a comment describing this scenario in the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sairameshv can you only call this
if podInfraContainer.Spoofed()?
otherwise, there will be two stopped events when the infra container is not dropped (one from handleExit, and one from here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah @haircommander , Updated.
|
/retest |
|
/retest |
littlejawa
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Thanks!
|
A friendly reminder that this PR had no activity for 30 days. |
|
/retest |
e1bd050 to
da5adfe
Compare
da5adfe to
c6aa5ae
Compare
@sohankunkerkar , Thanks for the reminder. I updated the PR today. Thanks !! |
|
/test e2e-gcp-ovn |
Fixes cri-o#6625 Signed-off-by: Sai Ramesh Vanka <[email protected]>
c6aa5ae to
30ddd6d
Compare
|
/approve LGTM, thanks @sairameshv , PTAL @cri-o/cri-o-maintainers |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: haircommander, littlejawa, sairameshv The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
/retest |
|
/test ci-cgroupv2-integration |
1 similar comment
|
/test ci-cgroupv2-integration |
|
/cherry-pick release-1.27 |
|
@haircommander: new pull request created: #6878 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Fixes #6625
Signed-off-by: Sai Ramesh Vanka [email protected]
What type of PR is this?
/kind bug
What this PR does / why we need it:
Which issue(s) this PR fixes:
#6625, #6631
Special notes for your reviewer:
Does this PR introduce a user-facing change?