Increase the concurrency share of workload-low priority level #95259
Conversation
|
Hi @tkashem. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I would like your thoughts on it, there might be some scenarios I am missing but it looks like the concurrency share of /assign @MikeSpreitzer @yue9944882 |
|
This PR may require API review. If so, when the changes are ready, complete the pre-review checklist and request an API review. Status of requested reviews is tracked in the API Review project. |
7d9fcfb to
fd7bf9a
Compare
|
/retest |
1 similar comment
|
/retest |
There was a problem hiding this comment.
This seems like a plausible improvement to me.
When designing API Priority and Fairness, we initially wanted to have some borrowing between priority levels. But it was a bit tricky to agree on a good definition for the behavior. We decided to do something simpler first, and later add that borrowing.
Please also note the following remark from https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190228-priority-and-fairness.md#non-goals :
This KEP does not introduce threading of additional information through webhooks and/or along other paths to support avoidance of priority inversions. While that is an attractive thing to consider in the future, this KEP is deliberately limited in its ambition. The intent for this KEP is to document that for the case of requests that are secondary to some other requests the configuration should identify those secondary requests and give them sufficiently high priority to avoid priority inversion problems. That will necessarily be approximate, and we settle for that now.
| Type: flowcontrol.PriorityLevelEnablementLimited, | ||
| Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ | ||
| AssuredConcurrencyShares: 20, | ||
| AssuredConcurrencyShares: 100, |
There was a problem hiding this comment.
The belief is that we have many more of these than we do workload-high clients?
There was a problem hiding this comment.
@deads2k global-default matches workloads that are running as non SA accounts. If I am not mistaken they are fewer compared to service-accounts which is basically all workloads (minus workload-high) running inside the cluster with service accounts.
service-accounts workload matches the workload-low priority level.
The concurrency pool of workload-low is 20 whereas that of global-default is 100, This PR swaps the values so service-accounts workloads have more concurrency share.
As far as workload-high is concerned, I have not seen workload-high workloads starve in my testing where we had about 13K Pods running at any time and a high namespace and pod churn rate. It was a 250-node cluster.
At the same time, I recently came across a CI job where workload-high was starving because a certain admission controller OwnerReferencesPermissionEnforcement was taking order of seconds to respond and triggering p&f to throttle workload-high.
|
/LGTM |
|
/ok-to-test |
|
/retest |
1 similar comment
|
/retest |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lavalamp, MikeSpreitzer, tkashem The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
1 similar comment
|
/retest |
|
/sig api-machinery |
|
/triage accepted |
|
/priority important-soon |
What type of PR is this?
/kind bug
What this PR does / why we need it:
All workloads running using service account (except for the ones distinguished by p&f with a logically higher matching precedence) will match the
service-accountsflow schema and be assigned to theworkload-lowpriority and thus will have only20concurrency shares. (~10% of the total)On the other hand,
global-defaultflow schema is assigned toglobal-defaultpriority configuration and thus will have100concurrency shares (~50% of the total). If I am not mistaken,global-defaultgoes pretty much unused since workloads running with user (not service account) will fall into this category and is not very common.Workload with service accounts do not have enough concurrency share and may starve. Increase the concurrency share of
workload-lowfrom20to100and reduce that ofglobal-defaultfrom100to20.Another potential solution: maybe when a concurrency pool is starving, it can borrow from another pool that has concurrency shares to spare?