kgo: revoke everything when the group protocol downgrades from cooperative to eager#1367
Open
twmb wants to merge 2 commits into
Open
kgo: revoke everything when the group protocol downgrades from cooperative to eager#1367twmb wants to merge 2 commits into
twmb wants to merge 2 commits into
Conversation
…ative to eager A group whose members are configured with both cooperative and eager balancers (the standard migration config) runs cooperatively until a member supporting only an eager protocol joins; the group's protocol vote is then forced to eager. We previously followed the flip with only a warning. The prior cooperative session revoked nothing at its end, while the eager session's diffAssigned returns the entire assignment as newly added: fetchOffsets then assigned offsets for partitions that were still actively fetching, giving each cursor two owners (the in-flight fetch and the new list/epoch load). The load's completion raced concurrent polls' cursor updates - a data race and a potentially torn cursor offset. The Java client pins each member's rebalance protocol at construction from its own assignor list, so a mixed-list Java member revokes everything on every rejoin and can never flip mid-life. We follow the group-selected protocol per generation - cooperative rebalancing whenever the whole group supports it - so we must handle the flip: on a downgrade, revoke all partitions before consuming the new assignment (invalidate everything, let the user commit in onRevoked, clear commit state), the state an eager session expects to begin from. A final autocommit between the downgrade sync and the revoke can land after another member fetched offsets; that yields duplicates only and cannot be closed client-side, since we only learn the selected protocol from the join response. Closes #1365
Regression for #1365: one member advertises [cooperative-sticky, sticky] and consumes hot while a second joins advertising only [sticky], forcing the group's protocol vote to sticky. Pre-fix this raced under -race in most runs; post-fix it is clean and every partition resumes consuming after the downgrade revoke.
92c5e7f to
bac25da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the data race in #1365 between
PollFetches(takeBuffered->finishUsingAllWithSet->cursor.setOffset) andhandleListOrEpochResults'cursor.setOffset.Root cause
A group whose members carry both cooperative and eager balancers (the standard migration config, e.g.
Balancers(CooperativeStickyBalancer(), RangeBalancer())) runs cooperatively until a member supporting only an eager protocol joins; the group's protocol vote is then forced to eager. The client followed the flip with only a warning:handleJoinRespflippedg.cooperativeto false.diffAssignedshort-circuits for eager consumers and returned the entire assignment as newly added, and prerevoke (gated on cooperative) invalidated nothing.fetchOffsets->assignPartitions(assignWithoutInvalidating)issued list/epoch loads (and direct offset sets) for partitions whose cursors were still fetching: two owners per cursor, and the load's completion raced concurrent polls' cursor updates.The Java client pins each member's rebalance protocol at construction from its own assignor list (a mixed-list Java member revokes everything on every rejoin, forever), so a live member there can never flip. franz-go follows the group-selected protocol per generation - cooperative rebalancing whenever the whole group supports it - so it uniquely owns this transition.
Fix
On a downgrade join, enter the eager era the way an eager session would have left the prior one, before offsets are fetched: invalidate all assignments, call
onRevokedwith the prior assignment (so final work can commit), and clear commit state. The old "not supported per KIP-429" warning was wrong per the KIP's downgrade section and is replaced with an INFO describing the revoke.A final autocommit between the downgrade sync and the revoke can land after another member fetched offsets; that window yields duplicates only (at-least-once holds) and cannot be closed client-side, since the selected protocol is only learned from the join response.
Verification
TestIssue1365CooperativeDowngrade): one member advertises[cooperative-sticky, sticky]and consumes hot while a second joins with only[sticky]. Pre-fix it reproduced the exact reported race pair in 3 of 5-raceruns; post-fix, 27/27 clean, and every partition resumes consuming after the downgrade revoke.-race -skip ETL: pass, zero races.-raceruns of pure cooperative-sticky churn confirm the invariant holds without a downgrade (the flip is the only feeder found; two independent full traces of the load/fetch ownership machinery agreed).#1271 and the "partition fetched while offsets concurrently fetched" oddity in #1222 share this signature and were both mixed-config-capable groups; this plausibly retires those as well.
Closes #1365