Fix vm containers could not restore after CRI-O restart #5574

sleepymole · 2022-01-26T13:25:35Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

When cri-o is running with kata runtime, cri-o could not restore containers correctly after it restarts. The containerd-shim-kata-v2 and qemu-system-x86_64 processes are leak every time.

This PR attempts to restore containers properly with the existing containerd-shim-kata-v2 and qemu-system-x86_64 processes. containerd-shim-kata-v2 writes the gRPC socket address under container bundle path with a file named address. After cri-o restarts, we can read from that file for shim socket address and connect to. Besides, we also need to restore the container io so that we can exec or attach the container.

I tested these changes on our internal cluster, everything worked well.

Which issue(s) this PR fixes:

Fixes #2112 #5569

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix vm containers couldn't restore after cri-o restart

openshift-ci · 2022-01-26T13:25:44Z

Hi @gozssky. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sleepymole · 2022-01-26T13:26:46Z

/cc @haircommander

internal/oci/runtime_vm.go

haircommander · 2022-01-26T13:40:48Z

internal/oci/runtime_vm.go

+	c.opLock.Lock()
+	defer c.opLock.Unlock()
+
+	// Lets ensure we're able to properly get construct the Options


this reads to me more of an operation to do when creating the runtime. WDYt about moving it there?

It seems ok, but this PR does n't change these lines.

ah nevermind then

This PR shuffles this code around, but this code was actually introduced to ensure we can configure the runtime to receive a specific configuration file.

internal/oci/runtime_vm.go

haircommander · 2022-01-26T13:43:21Z

internal/oci/runtime_vm.go

 		return errdefs.ErrNotFound
 	}

+	if err = r.restoreContainerIO(ctx, c, response); err != nil {


in the case where we haven't restarted, does this get called? should it?

yes, if container io has already existed in ctrs map, restoreContainerIO won't do more operations.

haircommander · 2022-01-26T13:44:29Z

/ok-to-test
/area vm

Awesome @gozssky, many thanks for taking this on! I have some nits and questions, but in general I like the approach.

@fidencio @fgiudici PTAL

codecov · 2022-01-26T13:50:53Z

Codecov Report

Merging #5574 (acde725) into main (025bebf) will decrease coverage by 0.17%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #5574      +/-   ##
==========================================
- Coverage   43.22%   43.05%   -0.18%     
==========================================
  Files         123      123              
  Lines       12214    12266      +52     
==========================================
+ Hits         5280     5281       +1     
- Misses       6426     6477      +51     
  Partials      508      508

haircommander · 2022-01-26T14:25:35Z

/retitle Fix vm containers could not restore after CRI-O restart

(ci doesn't handle apostrophes well...)

haircommander · 2022-01-26T14:25:44Z

/retest

fidencio · 2022-01-28T15:46:04Z

@gozssky, first and foremost, this PR works like a charm, thanks for your contribution!
Now, a few tips just to improve the life of the reviewer for next interactions:

In 61f9d86 you shuffle the CreateContainer around. I´d leave that out of that commit, as that drives our eyes for a change there, while that's not the important part of your PR.

Move that code around as part of the second patch, please, as you're already moving code around there.

You're pulling in ctrio as a new dependency, which I don't see as a problem. Historically, a long time ago, a bunch of containerd code was brought into CRI-O to avoid vendoring, which has proven to not be a good idea, so using it as a new dep as you did is the correct way. Still on this, could you also check whether we still need something from cio / cioutils? maybe everything we need already comes from ctrio and we could have a follow-up patch ensuring we use everything from there.

I'd appreciate if items 1 and 2 could be worked before getting this merged. 3 can come later.
Yet again, nice work @gozssky!

@fidencio Thanks for your review very much! I'm very sorry that I didn't check the code diff carefully before submitting this PR. But I'm not clear yet what I need to do for item 2. Could you explain a little more?

@gozssky, ah, sorry, just add the CreateContainer code move as part of the patch that's already moving things around.
That's it and it should be good to go! :-)

sleepymole · 2022-01-28T16:52:14Z

@fidencio I submitted the third patch 2e41e26. I'm not sure if I understand your suggestion well. 😅

fidencio · 2022-01-28T22:11:00Z

@gozssky, my suggestion:

On the commit 61f9d86, do not change the placement of the CreateContainer function.
Instead, do that as part of 9c21a51
Drop 2e41e26, and let's have that as a different PR.

Sounds reasonable?

Signed-off-by: Yujie Xia <[email protected]>

sleepymole · 2022-01-29T01:33:47Z

@gozssky, my suggestion:

On the commit 61f9d86, do not change the placement of the CreateContainer function.

Instead, do that as part of 9c21a51

Drop 2e41e26, and let's have that as a different PR.

Sounds reasonable?

@fidencio Thank you for the detailed explanation! I adjusted my two patches. The first patch 0731a9b didn't change the CreateContainer function. And the second patched acde725 reuse the createContainerIO logic in CreateContainer and move functions after the function that calls them. Since the commit hash changed, I force-push my two commits. I'm not sure if this is appropriate.

fidencio · 2022-01-29T09:43:01Z

@fidencio Thank you for the detailed explanation! I adjusted my two patches. The first patch 0731a9b didn't change the CreateContainer function. And the second patched acde725 reuse the createContainerIO logic in CreateContainer and move functions after the function that calls them. Since the commit hash changed, I force-push my two commits. I'm not sure if this is appropriate.

@gozssky, that's the way to go!
I will re-review / re-test it in the afternoon, thanks!

fidencio · 2022-01-29T09:43:20Z

/retest

fidencio · 2022-01-29T10:16:50Z

@gozssky, integration tests are breaking, please take a look at those:

not ok 74 ctr update resources
# (from function `copyimg' in file ./helpers.bash, line 234,
#  from function `setup_img' in file ./helpers.bash, line 245,
#  from function `setup_crio' in file ./helpers.bash, line 255,
#  from function `start_crio' in file ./helpers.bash, line 337,
#  in test file ./ctr.bats, line 729)
#   `start_crio' failed
# time="2022-01-29T09:50:35Z" level=error msg="Error importing dir:/home/runner/work/cri-o/cri-o/.artifacts/redis-image: writing blob: adding layer with blob \"sha256:276d6b52cd5b0a1ec792aff4b4feb6969e60985969775a009b277104b9532396\": Error processing tar file(exit status 127): Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!\n"
# time="2022-01-29T09:50:37Z" level=fatal msg="connect: connect endpoint 'unix:///tmp/tmp.r3tXFgBgkr/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"
# time="2022-01-29T09:50:39Z" level=fatal msg="connect: connect endpoint 'unix:///tmp/tmp.r3tXFgBgkr/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"

fgiudici

Nice patch, thanks @gozssky !
Cannot understand why tests fail: let me try to retest.

fgiudici · 2022-02-07T22:59:55Z

internal/oci/runtime_vm.go

+		}
+		return nil
+	}
+	_, err := r.createContainerIO(ctx, c, cio.WithFIFOs(ctrio.NewFIFOSet(cioCfg, closer)))


I don't like that much that we use the containerd/io package directly here (ctrio) while we have the cri-o/utils/io package that wraps the calls of the containerd/io one.
@fidencio, I have taken a look and It seems to me much easier to wrap the check and management of the FIFOs in the cri-o/utils/io package with something like a new "WithExistingFIFO" function rather than making explicit usage of the containerd/cio package all around.
It makes sense to address this in a following PR.

@fgiudici, I partially agree.

For a long time we decided to not directly vendor containerd code and we ended up actually copy and pasting code from containerd to our codebase, and as any copy & pasted code, those are not receiving bug fixes and are there to rot.
I'd be happy with directly vendor the code and get rid of everything we copied into our repos. That would be cleaner and would esnure we actually get bug fixes from those.

WDYT?

Ah, thanks @fidencio, now I see... we have copied the github.com/containerd/cri/pkg/server/io pkg in the cri-o/utils/io!
Vendoring that package would be way better (not looked at the dependencies chain btw), but may be cleaner to keep all the code of fifo management in only one package (maybe there is already something in the containerd/cri/pkg/server/io pkg).
This probably deserves a dedicated investigation and PR :-)

may be cleaner to keep all the code of fifo management in only one package

@fgiudici I couldn't agree more. I'd like to do this in next PR.

fgiudici · 2022-02-07T23:13:20Z

/retest

openshift-ci · 2022-02-09T06:26:51Z

@gozssky: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2	`acde725`	link	false	`/test e2e_cgroupv2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2022-02-09T15:02:53Z

/approve

I tag @fidencio for the final lgtm

openshift-ci · 2022-02-09T15:02:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gozssky, haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fidencio · 2022-02-10T15:34:26Z

I will run the last round of tests here and most likely have it merged sooner than later. :-)

fidencio · 2022-02-10T22:54:24Z

@gozssky, yet again, thanks a whole lot for the contribution!
/lgtm

openshift-bot · 2022-02-10T23:02:22Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

fidencio · 2022-02-14T09:14:54Z

/cherry-pick release-1.23

openshift-cherrypick-robot · 2022-02-14T09:15:42Z

@fidencio: new pull request created: #5633

Details

In response to this:

/cherry-pick release-1.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sleepymole requested review from fidencio, mrunalp and runcom as code owners January 26, 2022 13:25

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jan 26, 2022

openshift-ci bot requested review from giuseppe and nalind January 26, 2022 13:25

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 26, 2022

openshift-ci bot requested a review from haircommander January 26, 2022 13:26

sleepymole force-pushed the issue-2112 branch 2 times, most recently from 5458124 to 61f9d86 Compare January 26, 2022 13:34

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jan 26, 2022

sleepymole mentioned this pull request Jan 26, 2022

Add support for crio restart for RuntimeVM (v2) implementation #2112

Closed

haircommander reviewed Jan 26, 2022

View reviewed changes

internal/oci/runtime_vm.go Outdated Show resolved Hide resolved

haircommander reviewed Jan 26, 2022

View reviewed changes

internal/oci/runtime_vm.go Outdated Show resolved Hide resolved

haircommander reviewed Jan 26, 2022

View reviewed changes

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. area/vm Runtime VM related pull requests and issues and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 26, 2022

sleepymole force-pushed the issue-2112 branch from f162064 to 9c21a51 Compare January 26, 2022 14:06

openshift-ci bot changed the title ~~Fix vm containers couldn't restore after CRI-O restart~~ Fix vm containers could not restore after CRI-O restart Jan 26, 2022

fidencio mentioned this pull request Jan 28, 2022

persist shim sock path and use it for restore #4576

Closed

sleepymole added 2 commits January 29, 2022 09:11

Fix vm containers couldn't restore after CRI-O restart

0731a9b

Signed-off-by: Yujie Xia <[email protected]>

Reuse createContainerIO in CreateContainer

acde725

Signed-off-by: Yujie Xia <[email protected]>

sleepymole force-pushed the issue-2112 branch from 2e41e26 to acde725 Compare January 29, 2022 01:22

fgiudici reviewed Feb 7, 2022

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 9, 2022

openshift-ci bot assigned fidencio Feb 10, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 10, 2022

openshift-merge-robot merged commit d5f68d8 into cri-o:main Feb 11, 2022

sleepymole deleted the issue-2112 branch February 11, 2022 01:49

openshift-cherrypick-robot mentioned this pull request Feb 14, 2022

[release-1.23] Fix vm containers could not restore after CRI-O restart #5633

Merged

haircommander mentioned this pull request Feb 14, 2022

cri-o couldn't restore sandbox after restart #5569

Closed

fgiudici mentioned this pull request Feb 15, 2022

REQUEST: New membership for @fgiudici #5648

Closed

4 tasks

sleepymole mentioned this pull request Apr 17, 2022

runtime_vm: use containerd deps for container io directly #5806

Merged

Fix vm containers could not restore after CRI-O restart #5574

Fix vm containers could not restore after CRI-O restart #5574

Uh oh!

Conversation

sleepymole commented Jan 26, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Jan 26, 2022

Uh oh!

sleepymole commented Jan 26, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Jan 26, 2022

Uh oh!

codecov bot commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haircommander commented Jan 26, 2022

Uh oh!

haircommander commented Jan 26, 2022

Uh oh!

fidencio commented Jan 28, 2022

Uh oh!

sleepymole commented Jan 28, 2022

Uh oh!

fidencio commented Jan 28, 2022

Uh oh!

sleepymole commented Jan 29, 2022

Uh oh!

fidencio commented Jan 29, 2022

Uh oh!

fidencio commented Jan 29, 2022

Uh oh!

fidencio commented Jan 29, 2022

Uh oh!

fgiudici left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fgiudici commented Feb 7, 2022

Uh oh!

openshift-ci bot commented Feb 9, 2022

Uh oh!

haircommander commented Feb 9, 2022

Uh oh!

openshift-ci bot commented Feb 9, 2022

Uh oh!

fidencio commented Feb 10, 2022

Uh oh!

fidencio commented Feb 10, 2022

Uh oh!

openshift-bot commented Feb 10, 2022

Uh oh!

fidencio commented Feb 14, 2022

Uh oh!

codecov bot commented Jan 26, 2022 •

edited

Loading