Backoffs in Agent: #331
Merged
Backoffs in Agent: #331
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This mitigates for thundering herd issues. This is done in the agent and not in the transport so that backoffs in a transport can have their own logic. For example, in the gRPC transport, the server does the backoffs so the client doesn't need to. Signed-off-by: Jacob Weinstock <[email protected]>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #331 +/- ##
==========================================
+ Coverage 51.13% 51.24% +0.10%
==========================================
Files 105 105
Lines 6942 8870 +1928
==========================================
+ Hits 3550 4545 +995
- Misses 3127 4062 +935
+ Partials 265 263 -2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Jacob Weinstock <[email protected]>
There was a problem hiding this comment.
Pull Request Overview
This PR implements exponential backoff functionality in the agent to mitigate thundering herd issues when polling for work. The backoff is implemented at the agent level rather than transport level to allow different transports to handle their own retry logic.
- Adds exponential backoff with configurable max interval when no actions are available or errors occur
- Removes specific gRPC connection refused logging in favor of general backoff handling
- Adds command-line configuration for backoff max interval with 5-second default
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tink/agent/agent.go | Implements exponential backoff logic and configuration structure |
| tink/agent/internal/transport/grpc/grpc.go | Removes specific connection refused error logging |
| cmd/agent/main.go | Sets default backoff configuration |
| cmd/agent/flags.go | Adds command-line flag for backoff max interval |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Simpler. If more are needed they can be added in the future. Signed-off-by: Jacob Weinstock <[email protected]>
25210ef to
af3ab15
Compare
Also, git ignore vscode settings. Signed-off-by: Jacob Weinstock <[email protected]>
Signed-off-by: Jacob Weinstock <[email protected]>
03994ee to
08877d7
Compare
elauriault
pushed a commit
to elauriault/tinkerbell
that referenced
this pull request
Oct 17, 2025
## Description This mitigates for thundering herd issues. This is done in the agent and not in the transport, so that backoffs in a transport can have their logic. For example, in the gRPC transport, the server does the backoffs, so the client doesn't need to. ## Why is this needed Fixes: # ## How Has This Been Tested? ## How are existing users impacted? What migration steps/scripts do we need? ## Checklist: I have: - [ ] updated the documentation and/or roadmap (if required) - [ ] added unit or e2e tests - [ ] provided instructions on how to upgrade
elauriault
pushed a commit
to elauriault/tinkerbell
that referenced
this pull request
Oct 18, 2025
## Description This mitigates for thundering herd issues. This is done in the agent and not in the transport, so that backoffs in a transport can have their logic. For example, in the gRPC transport, the server does the backoffs, so the client doesn't need to. ## Why is this needed Fixes: # ## How Has This Been Tested? ## How are existing users impacted? What migration steps/scripts do we need? ## Checklist: I have: - [ ] updated the documentation and/or roadmap (if required) - [ ] added unit or e2e tests - [ ] provided instructions on how to upgrade
elauriault
pushed a commit
to elauriault/tinkerbell
that referenced
this pull request
Oct 18, 2025
## Description This mitigates for thundering herd issues. This is done in the agent and not in the transport, so that backoffs in a transport can have their logic. For example, in the gRPC transport, the server does the backoffs, so the client doesn't need to. ## Why is this needed Fixes: # ## How Has This Been Tested? ## How are existing users impacted? What migration steps/scripts do we need? ## Checklist: I have: - [ ] updated the documentation and/or roadmap (if required) - [ ] added unit or e2e tests - [ ] provided instructions on how to upgrade
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This mitigates for thundering herd issues. This is done in the agent and not in the transport, so that backoffs in a transport can have their logic. For example, in the gRPC transport, the server does the backoffs, so the client doesn't need to.
Why is this needed
Fixes: #
How Has This Been Tested?
How are existing users impacted? What migration steps/scripts do we need?
Checklist:
I have: