feat: check for clean shutdown on reboot #7190

mdsahiloss · 2023-08-04T18:02:12Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds a feature to verify at reboot time that shutdown was clean or not, If it was not clean then applies repair logic implemented here

Which issue(s) this PR fixes:

Fixes #7177

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Attempt to call c/storage Check() and Repair() when an unclean shutdown is detected

openshift-ci · 2023-08-04T18:02:28Z

Hi @MdSahil-oss. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

internal/lib/container_server.go

codecov · 2023-08-07T18:37:01Z

Codecov Report

Merging #7190 (60d50e2) into main (d4cb832) will decrease coverage by 0.07%.
Report is 6 commits behind head on main.
The diff coverage is 24.44%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7190      +/-   ##
==========================================
- Coverage   49.24%   49.18%   -0.07%     
==========================================
  Files         136      136              
  Lines       15517    15555      +38     
==========================================
+ Hits         7641     7650       +9     
- Misses       6972     6998      +26     
- Partials      904      907       +3

haircommander · 2023-08-07T19:53:24Z

the structure here looks good, I'm writing a test case to validate it, so let's
/hold
for that. Thanks @MdSahil-oss !

haircommander · 2023-08-08T15:30:03Z

/ok-to-test

haircommander · 2023-08-08T17:20:17Z

@MdSahil-oss can you cherry-pick haircommander@bb5c01e ? it passes here but fails on main :)

mdsahiloss · 2023-08-08T17:42:02Z

Sure @haircommander I can cherry pick haircommander@bb5c01e.

mdsahiloss · 2023-08-08T19:26:32Z

/ok-to-test

haircommander · 2023-08-08T19:42:40Z


go build  -trimpath  -ldflags '-s -w -X github.com/cri-o/cri-o/internal/version.buildDate='2023-08-08T19:35:28Z' ' -tags "selinux seccomp exclude_graphdriver_devicemapper exclude_graphdriver_btrfs containers_image_ostree_stub containers_image_openpgp" -o bin/crio github.com/cri-o/cri-o/cmd/crio
# github.com/cri-o/cri-o/internal/lib
internal/lib/container_server.go:116:23: undefined: errors.Join

compile error

internal/lib/container_server.go

mdsahiloss · 2023-08-09T07:45:15Z

/ok-to-test

haircommander · 2023-08-09T13:34:15Z

/approve

LGTM, thanks!
@cri-o/cri-o-maintainers PTAL

haircommander · 2023-08-09T13:36:42Z

/retest

internal/criocli/criocli.go

test/crio-wipe.bats

mdsahiloss · 2023-08-16T18:31:59Z

/retest

mdsahiloss · 2023-08-16T18:33:38Z

/retest

haircommander · 2023-08-17T18:36:23Z

@MdSahil-oss the test is failing because there are a handful more places you need to wire the configuration to. I have added two commits I think should be picked here and squashed into your other commits: https://github.com/haircommander/cri-o/tree/wipe-check-shut-down

haircommander · 2023-08-17T18:37:36Z

internal/criocli/criocli.go

 		},
+		&cli.BoolFlag{
+			Name:    "internal-repair",
+			Usage:   "If true, CRI-O will check if contents of an image was modified or removed and repair the image.",


nit: I think this should be

If true, CRI-O will check if the container and image storage was corrupted after a sudden restart, and attempt to repair the storage if it was.

@haircommander Done 👍🏽

mdsahiloss · 2023-08-17T19:18:07Z

/retest

haircommander · 2023-08-21T15:27:32Z

/retest

haircommander · 2023-08-21T16:50:48Z

Aug 21 15:45:53.149: INFO: Unexpected error occurred: rpc error: code = Unknown desc = container create failed: time="2023-08-21T15:45:53Z" level=error msg="runc create failed: unable to start container process: container init was OOM-killed (memory limit too low?)"

unexpected
/retest

mdsahiloss · 2023-08-21T18:36:24Z

/retest

…ot, If it was not clean then apply repair logic Signed-off-by: Md Sahil <[email protected]>

Signed-off-by: Md Sahil <[email protected]>

mdsahiloss · 2023-08-21T19:13:05Z

/retest

haircommander · 2023-08-21T19:55:21Z

/approve

LGTM, thanks @MdSahil-oss

PTAL @cri-o/cri-o-maintainers

openshift-ci · 2023-08-22T14:09:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, MdSahil-oss, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2023-08-22T14:25:14Z

/retest
/hold cancel

haircommander · 2023-08-22T15:04:37Z

/override ci/prow/ci-fedora-integration

openshift-ci · 2023-08-22T15:04:42Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/ci-fedora-integration

Details

In response to this:

/override ci/prow/ci-fedora-integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2023-08-22T18:16:25Z

/override ci/prow/ci-rhel-critest

openshift-ci · 2023-08-22T18:16:31Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/ci-rhel-critest

Details

In response to this:

/override ci/prow/ci-rhel-critest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mdsahiloss requested a review from mrunalp as a code owner August 4, 2023 18:02

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/feature Categorizes issue or PR as related to a new feature. labels Aug 4, 2023

openshift-ci bot requested review from QiWang19 and sohankunkerkar August 4, 2023 18:02

openshift-ci bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2023

sohankunkerkar changed the title ~~feat: Added a feature to check at reboot time shutdown was clean or n…~~ feat: check for clean shutdown on reboot Aug 4, 2023

haircommander reviewed Aug 7, 2023

View reviewed changes

internal/lib/container_server.go Outdated Show resolved Hide resolved

mdsahiloss force-pushed the wipe-check-shut-down branch from 624c7cf to 336abb7 Compare August 7, 2023 18:22

mdsahiloss force-pushed the wipe-check-shut-down branch from 336abb7 to 099322b Compare August 7, 2023 19:44

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2023

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 8, 2023

haircommander added this to the 1.28 milestone Aug 8, 2023

mdsahiloss requested a review from haircommander August 8, 2023 19:29

haircommander reviewed Aug 8, 2023

View reviewed changes

internal/lib/container_server.go Outdated Show resolved Hide resolved

mdsahiloss requested a review from haircommander August 9, 2023 07:41

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 9, 2023

haircommander reviewed Aug 16, 2023

View reviewed changes

internal/criocli/criocli.go Outdated Show resolved Hide resolved

haircommander reviewed Aug 16, 2023

View reviewed changes

test/crio-wipe.bats Outdated Show resolved Hide resolved

mdsahiloss force-pushed the wipe-check-shut-down branch from 1b41401 to be600a2 Compare August 16, 2023 18:33

haircommander reviewed Aug 17, 2023

View reviewed changes

mdsahiloss requested a review from haircommander August 18, 2023 17:13

mdsahiloss force-pushed the wipe-check-shut-down branch from 60d50e2 to 547509b Compare August 21, 2023 18:34

mdsahiloss added 2 commits August 22, 2023 00:39

feat: Added a feature to check at reboot time shutdown was clean or n…

39ea33e

…ot, If it was not clean then apply repair logic Signed-off-by: Md Sahil <[email protected]>

Added a flag internal-repair

7e3522a

Signed-off-by: Md Sahil <[email protected]>

mdsahiloss force-pushed the wipe-check-shut-down branch from 547509b to 7e3522a Compare August 21, 2023 19:12

saschagrunert approved these changes Aug 22, 2023

View reviewed changes

openshift-ci bot assigned saschagrunert Aug 22, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 22, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2023

openshift-merge-robot merged commit c4ffdbe into cri-o:main Aug 22, 2023

kwilczynski mentioned this pull request Jul 26, 2024

Use custom storage check options for CRI-O internal wipe #8417

Merged

feat: check for clean shutdown on reboot #7190

feat: check for clean shutdown on reboot #7190

Uh oh!

Conversation

mdsahiloss commented Aug 4, 2023 • edited by haircommander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Aug 4, 2023

Uh oh!

Uh oh!

codecov bot commented Aug 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haircommander commented Aug 7, 2023

Uh oh!

haircommander commented Aug 8, 2023

Uh oh!

haircommander commented Aug 8, 2023

Uh oh!

mdsahiloss commented Aug 8, 2023

Uh oh!

mdsahiloss commented Aug 8, 2023

Uh oh!

haircommander commented Aug 8, 2023

Uh oh!

Uh oh!

mdsahiloss commented Aug 9, 2023

Uh oh!

haircommander commented Aug 9, 2023

Uh oh!

haircommander commented Aug 9, 2023

Uh oh!

Uh oh!

Uh oh!

mdsahiloss commented Aug 16, 2023

Uh oh!

mdsahiloss commented Aug 16, 2023

Uh oh!

haircommander commented Aug 17, 2023

Uh oh!

haircommander Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

mdsahiloss Aug 19, 2023

Choose a reason for hiding this comment

Uh oh!

mdsahiloss commented Aug 17, 2023

Uh oh!

haircommander commented Aug 21, 2023

Uh oh!

haircommander commented Aug 21, 2023

Uh oh!

mdsahiloss commented Aug 21, 2023

Uh oh!

mdsahiloss commented Aug 21, 2023

Uh oh!

haircommander commented Aug 21, 2023

Uh oh!

openshift-ci bot commented Aug 22, 2023

Uh oh!

haircommander commented Aug 22, 2023

Uh oh!

haircommander commented Aug 22, 2023

Uh oh!

openshift-ci bot commented Aug 22, 2023

Uh oh!

haircommander commented Aug 22, 2023

Uh oh!

openshift-ci bot commented Aug 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

mdsahiloss commented Aug 4, 2023 •

edited by haircommander

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading