server: rework internal wipe to play better with reboots #4900
Conversation
40a3d8a to
7a926d4
Compare
|
/retest |
Before, we would not actually call CNI DEL on sandboxes that failed to restore. This was because we iterated through a map of *successfully* restored sandboxed, and checked if they failed. Instead, we need to rework LoadSandbox a bit to return a sandbox that was loaded. Even if the restore fails, we can pass the shell of a sandbox to the CNI plugin to best-effort cleanup Signed-off-by: Peter Hunt <[email protected]>
Signed-off-by: Peter Hunt <[email protected]>
Signed-off-by: Peter Hunt <[email protected]>
Signed-off-by: Peter Hunt <[email protected]>
7a926d4 to
1aaf474
Compare
|
/retest |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: haircommander, saschagrunert The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
@haircommander: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/lgtm |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
3 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/cherry-pick release-1.21 |
|
@haircommander: new pull request created: #4928 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
Embarrassingly, the main use-case of internal wipe was one that I had not tested appropriately. It turns out, we never actually have any containers wipeIfAppropriate after we call restore.
This is because the managed ns no longer exists (it is in /run, a tmpfs). Even if its not in a tmpfs, it is not correctly mounted on reboot.
this is okay, though. It reduces complexity a bit. Now, instead of using the version_file to determine whether we rebooted, we just have to make sure we call CNI DEL as much as possible if we fail to restore a sandbox.
To do this, we must rework LoadSandbox to return the sandbox in question (as soon as possible, basically after unmarshalling all the data from the config.json), then call networkStop on that sandbox (inside a resource cleaner so we retry if the node is booting up).
As a result of this, we can simplify wipeIfAppropriate. It now must be passed a list of images, but will not need to do any thinking about containers. Another consequence is we can eventually deprecate version_file.
This PR also ignores some error cases that were not useful to log.
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?