rate-limit events record in kubelet #13192
Conversation
|
Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist") If this message is too spammy, please complain to ixdy. |
3f7ec28 to
904a823
Compare
cmd/kubelet/app/server.go
Outdated
There was a problem hiding this comment.
Change help string to: "If > 0, limit event creations per second to this value. If 0, unlimited. [default=0.0]"
|
We have default rate-limiting in the client. Should document how that interacts with this event QPS limit. Like, I think right now if your client has a limit of 5 QPS and your events have a limit of 10 QPS, then effectively events are limited to 5 QPS and everything else could be starved. I'm not sure if this is a feature or a bug. |
5d5cbe6 to
9c6cbbb
Compare
|
@lavalamp Thanks a lot for your review and advise. I made changes as you said and I'm not sure about where to document interacts of event QPS and client QPS. Should we suggest event QPS lower than client's default limit of 5 QPS? |
|
Labelling this PR as size/L |
cmd/kubelet/app/server.go
Outdated
There was a problem hiding this comment.
Terribly sorry that I've been sitting on this, I guess I didn't see that you pushed changes. I have been thinking about this for a bit, and it occurs to me that there's another way to do this.
The solution you have here will work, but there's two downsides:
- People pass
nileverywhere now. - It's unpredictable how much quota will be left for non-event client usage.
An idea that fixes both of these is:
- Make a copy of kcfg.KubeClient
- Modify the copy's internal RESTClient.Throttle object (I think enough is public that you can do this)
- Start recording to this copied client.
Then you don't have to add throttling to the event recorder at all, and the event QPS won't count against the regular QPS.
I'm sorry for not thinking of this to begin with. What do you think?
There was a problem hiding this comment.
@lavalamp Thanks a lot for your idea. Totally agree with you. I think this will make event recorder behavior more clear. Will update this pr soon.
1. Add EvnetRecordQps and EventBurst parameter in kubelet. 2. If EvnetRecordQps and EventBurst was set, rate limit events in kubelet with a independent ratelimiter as setted.
9c6cbbb to
1460a1f
Compare
|
Labelling this PR as size/XL |
|
GCE e2e build/test passed for commit 1460a1f. |
There was a problem hiding this comment.
Hm, I'm puzzled about why github shows such a bad diff on this file. Weird.
|
This looks great, thank you very much for bearing with me! LGTM |
|
The author of this PR is not in the whitelist for merge, can one of the admins add the 'ok-to-merge' label? |
|
@k8s-bot test this [submit-queue is verifying that this PR is safe to merge] |
|
GCE e2e build/test passed for commit 1460a1f. |
|
Automatic merge from SubmitQueue |
Auto commit by PR queue bot
…-#13192-upstream-release-1.0 Automated cherry pick of #13192
Fixes #11912