kgo: fix for empty fetch mapped metadata #1143
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
r-vasquez
reviewed
Oct 20, 2025
pkg/kfake/issues_test.go
Outdated
Comment on lines
585
to
593
| if resp.Cluster != "kfake" { | ||
| t.Fatalf("expected cluster kfake, got %s", resp.Cluster) | ||
| } | ||
| if resp.Controller != 0 { | ||
| t.Fatalf("expected controller 0, got %d", resp.Controller) | ||
| } | ||
| if len(resp.Brokers) != 1 { | ||
| t.Fatalf("expected 1 broker, got %d", len(resp.Brokers)) | ||
| } |
Contributor
There was a problem hiding this comment.
Can we add a test for the topics returned? something like
if len(resp.Topics.Names()) != 1 {
t.Fatalf("expected exactly one topic; got: %d", len(resp.Topics.Names()))
}
Right now we are seeing an empty topic list, mainly because we are passing an empty slice (instead of nil) to the metadata request.
I believe this should fix it unless there is a better way to determine when we need to fetch for all topics when the cache is empty:
diff --git a/pkg/kgo/client.go b/pkg/kgo/client.go
index 60e2a907..495a4430 100644
--- a/pkg/kgo/client.go
+++ b/pkg/kgo/client.go
@@ -1296,7 +1296,7 @@ func (cl *Client) Request(ctx context.Context, req kmsg.Request) (kmsg.Response,
// set to true. This function cannot be used to request topics via TopicID;
// the direct topic name must be used.
func (cl *Client) RequestCachedMetadata(ctx context.Context, req *kmsg.MetadataRequest, limit time.Duration) (*kmsg.MetadataResponse, e
rror) {
- topics := make([]string, 0, len(req.Topics))
+ var topics []string
for _, t := range req.Topics {
if t.Topic == nil || *t.Topic == "" {
return nil, errors.New("unable to request cached metadata with a missing topic name (topic IDs are not supported
)")
@@ -2690,7 +2690,7 @@ func (cl *Client) fetchMappedMetadata(ctx context.Context, topics []string, useC
intoMapped = make(map[string]mappedMetadataTopic)
}
- _, _, err := cl.fetchMetadataForTopics(ctx, false, needed, intoMapped)
+ _, _, err := cl.fetchMetadataForTopics(ctx, topics == nil, needed, intoMapped)
return intoMapped, err
}
Owner
There was a problem hiding this comment.
Just to be a little bit more paranoid about the logic here, wdyt of this diff?
func (cl *Client) RequestCachedMetadata(ctx context.Context, req *kmsg.MetadataRequest, limit time.Duration) (*kmsg.MetadataResponse, error) {
- topics := make([]string, 0, len(req.Topics))
+ var topics []string
+ if req.Topics != nil {
+ topics = make([]string, 0, len(req.Topics))
+ }
for _, t := range req.Topics {
if t.Topic == nil || *t.Topic == "" {
return nil, errors.New("unable to request cached metadata with a missing topic name (topic IDs are not supported)")
@@ -2652,7 +2660,7 @@ func (cl *Client) maybeDeleteMappedMetadata(unknownTopic bool, ts ...string) (sh
func (cl *Client) fetchCachedMappedMetadata(limit time.Duration, ts ...string) (map[string]mappedMetadataTopic, []string) {
cl.mappedMetaMu.Lock()
defer cl.mappedMetaMu.Unlock()
- if cl.mappedMeta == nil {
+ if len(cl.mappedMeta) == 0 {
return nil, ts
}
cached := make(map[string]mappedMetadataTopic)
@@ -2682,7 +2690,12 @@ func (cl *Client) fetchMappedMetadata(ctx context.Context, topics []string, useC
needed := topics
if useCache {
intoMapped, needed = cl.fetchCachedMappedMetadata(limit, topics...)
- if len(needed) == 0 {
+ // If intoMapped is nil, we have no cached topics at all and
+ // need to force a metadata load to satisfy broker/controller
+ // aspects of the metadata response. We have either never
+ // issued a metadata request, or the cached data (of no topics)
+ // could be super old. Cache age is only tracked per topic.
+ if intoMapped != nil && len(needed) == 0 {
return intoMapped, nil
}
}
@@ -2690,7 +2703,7 @@ func (cl *Client) fetchMappedMetadata(ctx context.Context, topics []string, useC
intoMapped = make(map[string]mappedMetadataTopic)
}
- _, _, err := cl.fetchMetadataForTopics(ctx, false, needed, intoMapped)
+ _, _, err := cl.fetchMetadataForTopics(ctx, topics == nil, needed, intoMapped)
return intoMapped, err
}The difference is:
- First off, adds a big doc comment on why this logic exists
- Sets topics to non-nil if the request topics is non-nil even if the len is 0
- Returns nil if mappedMeta len is 0 -- which covers the case for if we requested broker-only metadata 1hr ago, and want to re-request it now (i.e.: always bypass the cache if the cache is empty)
The only thing missing is that req.Version == 0 should request metadata when the topics are nil or empty, but I think people aren't going to be setting the request version in the standard use case of RequestCachedMetadata... so I'm fine to leave that out (which might break Kafka 0.8 and 0.9 users, who should upgrade).
e64d9d2 to
f770a7a
Compare
Owner
|
Left a comment -- good find. Mind splitting the kfake test into a different PR, and dropping the go.mod replace changes? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
intoMappednil, fetch fetch metadata when there is nothing set in the cache 🤞🏻Hopefully the test and gomod changes are ok, otherwise I'll revert it 👍🏻
cc #1142