fix(discov): add retry cooldown to prevent CPU/disk exhaustion on auth errors #5347
+7
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…h errors Fixes zeromicro#3823 When etcd authentication fails with 'invalid auth token' error, the watch() function was entering an infinite tight loop without any delay, causing: - 100% CPU usage - Gigabytes of identical error logs in seconds - Server crashes due to resource exhaustion Added a 1-second cooldown between retries using coolDownUnstable.AroundDuration() to prevent this issue while maintaining retry capability for transient errors. This approach is consistent with existing patterns in the codebase and adds slight randomness (±5% deviation) to prevent thundering herd scenarios.
9b3bf8f to
c43c7f1
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses issue 3823 by making documentation improvements and adding resilience protection to the etcd watch retry loop.
- Condenses and improves readability of README files (both English and Chinese versions)
- Adds cooldown mechanism to etcd watch error retry loop to prevent CPU/disk resource exhaustion
- Makes the AI tooling setup instructions more concise
Reviewed changes
Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| readme.md | Condensed background, design principles, features, and AI tooling sections for better readability; removed verbose explanations in favor of concise bullet points |
| readme-cn.md | Similar condensing and formatting improvements as English version, maintaining consistency between both language versions |
| core/discov/internal/registry.go | Added time.Sleep with cooldown after error logging in the watch function retry loop to prevent rapid retries from exhausting CPU/disk resources; follows the existing pattern used in the load function |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #3823
When etcd authentication fails with 'invalid auth token' error, the watch()
function was entering an infinite tight loop without any delay, causing:
Added a 1-second cooldown between retries using coolDownUnstable.AroundDuration()
to prevent this issue while maintaining retry capability for transient errors.
This approach is consistent with existing patterns in the codebase and adds
slight randomness (±5% deviation) to prevent thundering herd scenarios.