-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Introduce InternalWipe #4767
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
openshift-merge-robot
merged 20 commits into
cri-o:master
from
haircommander:more-cni-del
May 8, 2021
Merged
Introduce InternalWipe #4767
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
996508d
server: use background context for network stop
haircommander 6568e99
Cleanup pod network on sandbox removal
saschagrunert bccccf1
server: reuse container removal code for infra
haircommander 3c418d5
storage: remove RemovePodSandbox function
haircommander a627960
server: breakup stop/remove all functions with internal helpers
haircommander 3b4b5d6
config: add InternalWipe
haircommander 230eeac
crio wipe: add support for internal_wipe
haircommander 7ae3930
server: add support for internal_wipe
haircommander f0d91a2
test: add test for internal_wipe
haircommander d060232
Add resource cleaner retry functionality
saschagrunert 87148ff
server: move newPodNetwork to a more logical place
haircommander d243f6d
server: get hooks after we've check if a sandbox is already stopped
haircommander 1fcca60
InternalWipe: retry on failures
haircommander ce6527b
test: add test for delayed cleanup of network on restart
haircommander 9e15b55
resourcestore: run cleanup in parallel
haircommander 02c64b7
server: group namespace cleanup with network stop
haircommander b2eb270
server: don't unconditionally fail on sandbox cleanup
haircommander 72385f4
sandbox: fix race with cleanup
haircommander 616ae87
use more ContainerServer.StopContainer
haircommander 20354c7
sandbox remove: unmount shm before removing infra container
haircommander File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,33 +1,85 @@ | ||
| package resourcestore | ||
|
|
||
| // A CleanupFunc is a function that cleans up one piece of | ||
| // the associated resource. | ||
| type CleanupFunc func() | ||
| import ( | ||
| "context" | ||
| "time" | ||
|
|
||
| "github.com/pkg/errors" | ||
| "k8s.io/apimachinery/pkg/util/wait" | ||
|
|
||
| "github.com/cri-o/cri-o/internal/log" | ||
| ) | ||
|
|
||
| // ResourceCleaner is a structure that tracks | ||
| // how to cleanup a resource. | ||
| // CleanupFuncs can be added to it, and it can be told to | ||
| // Cleanup the resource | ||
| type ResourceCleaner struct { | ||
| funcs []CleanupFunc | ||
| funcs []cleanupFunc | ||
| } | ||
|
|
||
| // A cleanupFunc is a function that cleans up one piece of | ||
| // the associated resource. | ||
| type cleanupFunc func() error | ||
|
|
||
| // NewResourceCleaner creates a new ResourceCleaner | ||
| func NewResourceCleaner() *ResourceCleaner { | ||
| return &ResourceCleaner{ | ||
| funcs: make([]CleanupFunc, 0), | ||
| } | ||
| return &ResourceCleaner{} | ||
| } | ||
|
|
||
| // Add adds a new CleanupFunc to the ResourceCleaner | ||
| func (r *ResourceCleaner) Add(f func()) { | ||
| r.funcs = append(r.funcs, CleanupFunc(f)) | ||
| func (r *ResourceCleaner) Add( | ||
| ctx context.Context, | ||
| description string, | ||
| fn func() error, | ||
| ) { | ||
| // Create a retry task on top of the provided function | ||
| task := func() error { | ||
| err := retry(ctx, fn) | ||
| if err != nil { | ||
| log.Errorf(ctx, | ||
| "Retried cleanup function %q too often, giving up", | ||
| description, | ||
| ) | ||
| } | ||
| return err | ||
| } | ||
|
|
||
| // Prepend reverse iterate by default | ||
| r.funcs = append([]cleanupFunc{task}, r.funcs...) | ||
| } | ||
|
|
||
| // Cleanup cleans up the resource, running | ||
| // the cleanup funcs in opposite chronological order | ||
| func (r *ResourceCleaner) Cleanup() { | ||
| for i := len(r.funcs) - 1; i >= 0; i-- { | ||
| r.funcs[i]() | ||
| func (r *ResourceCleaner) Cleanup() error { | ||
| for _, f := range r.funcs { | ||
| if err := f(); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // retry attempts to execute fn up to defaultRetryTimes if its failure meets | ||
| // retryCondition. | ||
| func retry(ctx context.Context, fn func() error) error { | ||
| backoff := wait.Backoff{ | ||
| Duration: 500 * time.Millisecond, | ||
| Factor: 1.5, | ||
| Steps: defaultRetryTimes, | ||
| } | ||
|
|
||
| waitErr := wait.ExponentialBackoff(backoff, func() (bool, error) { | ||
| if err := fn(); err != nil { | ||
| log.Errorf(ctx, "Failed to cleanup (probably retrying): %v", err) | ||
| return false, nil | ||
| } | ||
| return true, nil | ||
| }) | ||
|
|
||
| if waitErr != nil { | ||
| return errors.Wrap(waitErr, "wait on retry") | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| // +build !test | ||
|
|
||
| package resourcestore | ||
|
|
||
| // defaultRetryTimes defines the amount of default retries for each cleanup | ||
| // function. | ||
| var defaultRetryTimes = 20 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| package resourcestore_test | ||
|
|
||
| import ( | ||
| "errors" | ||
|
|
||
| "github.com/cri-o/cri-o/internal/resourcestore" | ||
| . "github.com/onsi/ginkgo" | ||
| . "github.com/onsi/gomega" | ||
| "golang.org/x/net/context" | ||
| ) | ||
|
|
||
| // The actual test suite | ||
| var _ = t.Describe("ResourceCleaner", func() { | ||
| It("should call the cleanup functions", func() { | ||
| // Given | ||
| sut := resourcestore.NewResourceCleaner() | ||
| called1 := false | ||
| called2 := false | ||
| sut.Add(context.Background(), "test1", func() error { | ||
| called1 = true | ||
| return nil | ||
| }) | ||
| sut.Add(context.Background(), "test2", func() error { | ||
| called2 = true | ||
| return nil | ||
| }) | ||
|
|
||
| // When | ||
| err := sut.Cleanup() | ||
|
|
||
| // Then | ||
| Expect(err).To(BeNil()) | ||
| Expect(called1).To(BeTrue()) | ||
| Expect(called2).To(BeTrue()) | ||
| }) | ||
|
|
||
| It("should retry the cleanup functions", func() { | ||
| // Given | ||
| sut := resourcestore.NewResourceCleaner() | ||
| called1 := false | ||
| called2 := false | ||
| sut.Add(context.Background(), "test1", func() error { | ||
| called1 = true | ||
| return nil | ||
| }) | ||
| failureCnt := 0 | ||
| sut.Add(context.Background(), "test2", func() error { | ||
| if failureCnt == 2 { | ||
| called2 = true | ||
| return nil | ||
| } | ||
| failureCnt++ | ||
| return errors.New("") | ||
| }) | ||
|
|
||
| // When | ||
| err := sut.Cleanup() | ||
|
|
||
| // Then | ||
| Expect(err).To(BeNil()) | ||
| Expect(called1).To(BeTrue()) | ||
| Expect(called2).To(BeTrue()) | ||
| Expect(failureCnt).To(Equal(2)) | ||
| }) | ||
|
|
||
| It("should retry three times", func() { | ||
| // Given | ||
| sut := resourcestore.NewResourceCleaner() | ||
| failureCnt := 0 | ||
| sut.Add(context.Background(), "test", func() error { | ||
| failureCnt++ | ||
| return errors.New("") | ||
| }) | ||
|
|
||
| // When | ||
| err := sut.Cleanup() | ||
|
|
||
| // Then | ||
| Expect(err).NotTo(BeNil()) | ||
| Expect(failureCnt).To(Equal(3)) | ||
| }) | ||
|
|
||
| It("should run in parallel", func() { | ||
| // Given | ||
| sut := resourcestore.NewResourceCleaner() | ||
| testChan := make(chan bool, 1) | ||
| succ := false | ||
| sut.Add(context.Background(), "test1", func() error { | ||
| testChan <- true | ||
| return nil | ||
| }) | ||
| sut.Add(context.Background(), "test2", func() error { | ||
| <-testChan | ||
| succ = true | ||
| return nil | ||
| }) | ||
|
|
||
| // When | ||
| err := sut.Cleanup() | ||
|
|
||
| // Then | ||
| Expect(err).To(BeNil()) | ||
| Expect(succ).To(BeTrue()) | ||
| }) | ||
| }) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.