Change default attach_detach_controller reconciler sync period to 1 minute#41363
Merged
k8s-github-robot merged 1 commit intoFeb 14, 2017
Merged
Conversation
When default reconciler sync period is set to 5 second, we often see rateLimit issue for a large cluster. This PR is change the period to 1 minute to mitigate this problem. Make this period longer means that there might be some period of time that the cached information in master's attach_detach_controller is out of date. The node might use this information to mount to the wrong device. For GCE PD, since device path is uniquely associated with volume id, so mount operation will just fail because of this outdated information. For AWS, before kubelet might mount to the wrong volume because device path could be reused immediately once it is available. But after PR kubernetes#38818, device path will only be reused after all device paths have been explored. That means it is very unlikely that kubelet will mount to a wrong volume that is using the old device path that had been assigned to the same node.
|
This change is |
Contributor
Author
|
/approve |
1 similar comment
Member
|
/approve |
Member
|
/lgtm |
Member
|
/approve |
gnufied
reviewed
Feb 13, 2017
| ClusterSigningCertFile: "/etc/kubernetes/ca/ca.pem", | ||
| ClusterSigningKeyFile: "/etc/kubernetes/ca/ca.key", | ||
| ReconcilerSyncLoopPeriod: metav1.Duration{Duration: 5 * time.Second}, | ||
| ReconcilerSyncLoopPeriod: metav1.Duration{Duration: 60 * time.Second}, |
Member
There was a problem hiding this comment.
The default 5 second also has a problem that - polling an entire cluster one by one (like how we do currently) is going to take more than 5 seconds even if done parallely, so likely we have overlapping polls going on.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED The following people have approved this PR: gnufied, jingxu97, saad-ali Needs approval from an approver in each of these OWNERS Files: We suggest the following people: |
Member
|
@k8s-bot gci gke e2e test this |
Member
|
@k8s-bot unit test this |
Contributor
Author
|
@k8s-bot gce etcd3 e2e test this |
|
Automatic merge from submit-queue |
|
Commit found in the "release-1.5" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked. |
This was referenced Feb 14, 2017
jessfraz
added a commit
that referenced
this pull request
Feb 22, 2017
…63-upstream-release-1.4 Automated cherry pick of #41363
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When default reconciler sync period is set to 5 second, we often see
rateLimit issue for a large cluster. This PR is changing the period to 1
minute to mitigate this problem.
Make this period longer means that there might be some period of time
that the cached information in master's attach_detach_controller is out
of date. The node might use this information to mount to the wrong
device. For GCE PD, since device path is uniquely associated with volume
id, so mount operation will just fail because of this outdated
information. For AWS, before kubelet might mount to the wrong volume
because device path could be reused immediately once it is available.
But after PR #38818, device path will only be reused after all device
paths have been explored. That means it is very unlikely that kubelet will
mount to a wrong volume that is using the old device path that had been
assigned to the same node.
Release note: