[1.23] crio:fix a bug about log container #6409

laxmanvallandas · 2022-11-30T18:31:51Z

What type of PR is this?

/kind bug

What happened?

Pod stuck in terminating state indefinitely until the kubelet is restarted as the crio-o responds with below response during the deletion process.

Type     Reason      Age                    From     Message
  ----     ------      ----                   ----     -------
  Warning  FailedSync  4m35s (x2331 over 8h)  kubelet  error determining status: status.CreatedAt is not set

This has larger impact when the nodes are being updated(or auto-scaled to low count) and they can't just go down as the pod is still at terminating state.

Nov 30 03:38:02  kubelet[5584]: I1130 03:38:02.145598    5584 kubelet.go:2142] "SyncLoop (PLEG): event for pod" pod="redacted POD NAME" event=&{ID:d522de36-73e7-4a24-a087-2a1dc4f85c31 Type:ContainerStarted Data:redacted}
Nov 30 03:38:02  kubelet[5584]: E1130 03:38:02.147723    5584 remote_runtime.go:343] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="redacted"
Nov 30 03:38:02  systemd[266494]: Reached target Paths.
Nov 30 03:38:02 systemd[266494]: Starting D-Bus User Message Bus Socket.
Nov 30 03:38:02 systemd[266494]: Reached target Timers.
Nov 30 03:38:02  kubelet[5584]: E1130 03:38:02.643292    5584 pod_workers.go:951] "Error syncing pod, skipping" err="status.CreatedAt is not set" pod="redacted POD NAME" podUID=redacted

What did you expect to happen?

Crio should not return error while terminating pod.

How can we reproduce it (as minimally and precisely as possible)?

Not totally sure. It happens intermittently and not all pods deleted are in this state.

Anything else we need to know?

Fix provided here seem to have solved the issue. But it is available in 1.24 version and we would like to back-port it to 1.23 version in order to survive without noise.
Hence, this request to cherry pick the fix into 1.23.

CRI-O and Kubernetes version
Details
$ crio --version
crio version 1.22.5
Version:          1.22.5
GitCommit:        bd7ea645cd7436edd6a5a07b11559906032f124f
GitTreeState:     clean
BuildDate:        2022-07-29T04:19:32Z
GoVersion:        go1.16.12
Compiler:         gc
Platform:         linux/amd64
Linkmode:         dynamic
[root@ip-10-117-140-141 ~]#
$ kubectl --version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.14", GitCommit:"3321ffc07d2f046afdf613796f9032f4460de093", GitTreeState:"clean", BuildDate:"2022-11-09T13:40:19Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.14", GitCommit:"3321ffc07d2f046afdf613796f9032f4460de093", GitTreeState:"clean", BuildDate:"2022-11-09T13:32:47Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}```
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
$ uname -a
Linux ip-10-117-140-141.us-west-2.compute.internal 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Tue Sep 13 18:09:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Does this PR introduce a user-facing change?

Fix a bug about log container

Additional environment details (AWS, VirtualBox, physical, etc.)

Details

Cluster is running on AWS

openshift-ci · 2022-11-30T18:32:07Z

Hi @laxmanvallandas. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2022-11-30T18:34:50Z

thanks for opening the PR! I see you chose the merge commit instead of the original commit. I think the DCO is mad bc there's no sign-off on the merge commit. Would you mind replacing the commit you've chosen with 9f6a672?

haircommander · 2022-11-30T18:34:59Z

/ok-to-test

codecov · 2022-11-30T20:10:48Z

Codecov Report

Merging #6409 (4d9f768) into release-1.23 (a6a1e6e) will increase coverage by 0.03%.
The diff coverage is 100.00%.

❗ Current head 4d9f768 differs from pull request most recent head be5c64c. Consider uploading reports for the commit be5c64c to get more accurate results

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.23    #6409      +/-   ##
================================================
+ Coverage         42.98%   43.02%   +0.03%     
================================================
  Files               123      123              
  Lines             12399    12396       -3     
================================================
+ Hits               5330     5333       +3     
+ Misses             6560     6556       -4     
+ Partials            509      507       -2

Signed-off-by: jiang wei <[email protected]>

laxmanvallandas · 2022-12-01T08:59:45Z

Would you mind replacing the commit you've chosen with 9f6a672?

@haircommander , re-pushed using origin commit.

haircommander · 2022-12-01T14:03:08Z

/retitle [1.23] crio:fix a bug about log container
/approve
@saschagrunert PTAL

saschagrunert

/lgtm

openshift-ci · 2022-12-01T14:30:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, laxmanvallandas, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

laxmanvallandas · 2022-12-27T13:36:31Z

@haircommander Can this be merged?

haircommander · 2023-01-03T17:35:27Z

/release-note-none

haircommander · 2023-01-03T18:43:52Z

/retest

laxmanvallandas requested review from mrunalp and runcom as code owners November 30, 2022 18:31

openshift-ci bot added dco-signoff: no Indicates the PR's author has not DCO signed all their commits. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 30, 2022

openshift-ci bot requested review from haircommander and rhatdan November 30, 2022 18:32

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 30, 2022

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 30, 2022

crio:fix a bug about log container

be5c64c

Signed-off-by: jiang wei <[email protected]>

laxmanvallandas force-pushed the release-1.23 branch from 6930988 to be5c64c Compare December 1, 2022 08:47

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Dec 1, 2022

openshift-ci bot changed the title ~~Merge pull request #5843 from jwcesign/main~~ [1.23] crio:fix a bug about log container Dec 1, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 1, 2022

saschagrunert approved these changes Dec 1, 2022

View reviewed changes

openshift-ci bot assigned saschagrunert Dec 1, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 1, 2022

openshift-ci bot removed the release-note-none Denotes a PR that doesn't merit a release note. label Jan 3, 2023

haircommander removed the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 3, 2023

openshift-ci bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 3, 2023

openshift-merge-robot merged commit 2ff6a26 into cri-o:release-1.23 Jan 3, 2023

laxmanvallandas deleted the release-1.23 branch January 3, 2023 20:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[1.23] crio:fix a bug about log container #6409

[1.23] crio:fix a bug about log container #6409

Uh oh!

laxmanvallandas commented Nov 30, 2022 •

edited by haircommander

Loading

Uh oh!

openshift-ci bot commented Nov 30, 2022

Uh oh!

haircommander commented Nov 30, 2022

Uh oh!

haircommander commented Nov 30, 2022

Uh oh!

codecov bot commented Nov 30, 2022 •

edited

Loading

Uh oh!

laxmanvallandas commented Dec 1, 2022

Uh oh!

haircommander commented Dec 1, 2022

Uh oh!

saschagrunert left a comment

Uh oh!

openshift-ci bot commented Dec 1, 2022

Uh oh!

laxmanvallandas commented Dec 27, 2022

Uh oh!

haircommander commented Jan 3, 2023

Uh oh!

haircommander commented Jan 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[1.23] crio:fix a bug about log container #6409

[1.23] crio:fix a bug about log container #6409

Uh oh!

Conversation

laxmanvallandas commented Nov 30, 2022 • edited by haircommander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Does this PR introduce a user-facing change?

Additional environment details (AWS, VirtualBox, physical, etc.)

Uh oh!

openshift-ci bot commented Nov 30, 2022

Uh oh!

haircommander commented Nov 30, 2022

Uh oh!

haircommander commented Nov 30, 2022

Uh oh!

codecov bot commented Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

laxmanvallandas commented Dec 1, 2022

Uh oh!

haircommander commented Dec 1, 2022

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Dec 1, 2022

Uh oh!

laxmanvallandas commented Dec 27, 2022

Uh oh!

haircommander commented Jan 3, 2023

Uh oh!

haircommander commented Jan 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

laxmanvallandas commented Nov 30, 2022 •

edited by haircommander

Loading

codecov bot commented Nov 30, 2022 •

edited

Loading