Thanks to visit codestin.com
Credit goes to github.com

Skip to content

kgo: revoke everything when the group protocol downgrades from cooperative to eager#1367

Open
twmb wants to merge 2 commits into
masterfrom
kgo-cooperative-downgrade
Open

kgo: revoke everything when the group protocol downgrades from cooperative to eager#1367
twmb wants to merge 2 commits into
masterfrom
kgo-cooperative-downgrade

Conversation

@twmb

@twmb twmb commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Fixes the data race in #1365 between PollFetches (takeBuffered -> finishUsingAllWithSet -> cursor.setOffset) and handleListOrEpochResults' cursor.setOffset.

Root cause

A group whose members carry both cooperative and eager balancers (the standard migration config, e.g. Balancers(CooperativeStickyBalancer(), RangeBalancer())) runs cooperatively until a member supporting only an eager protocol joins; the group's protocol vote is then forced to eager. The client followed the flip with only a warning:

  1. The prior cooperative session revoked nothing at its end (correct for cooperative).
  2. handleJoinResp flipped g.cooperative to false.
  3. diffAssigned short-circuits for eager consumers and returned the entire assignment as newly added, and prerevoke (gated on cooperative) invalidated nothing.
  4. fetchOffsets -> assignPartitions(assignWithoutInvalidating) issued list/epoch loads (and direct offset sets) for partitions whose cursors were still fetching: two owners per cursor, and the load's completion raced concurrent polls' cursor updates.

The Java client pins each member's rebalance protocol at construction from its own assignor list (a mixed-list Java member revokes everything on every rejoin, forever), so a live member there can never flip. franz-go follows the group-selected protocol per generation - cooperative rebalancing whenever the whole group supports it - so it uniquely owns this transition.

Fix

On a downgrade join, enter the eager era the way an eager session would have left the prior one, before offsets are fetched: invalidate all assignments, call onRevoked with the prior assignment (so final work can commit), and clear commit state. The old "not supported per KIP-429" warning was wrong per the KIP's downgrade section and is replaced with an INFO describing the revoke.

A final autocommit between the downgrade sync and the revoke can land after another member fetched offsets; that window yields duplicates only (at-least-once holds) and cannot be closed client-side, since the selected protocol is only learned from the join response.

Verification

  • New kfake regression test (TestIssue1365CooperativeDowngrade): one member advertises [cooperative-sticky, sticky] and consumes hot while a second joins with only [sticky]. Pre-fix it reproduced the exact reported race pair in 3 of 5 -race runs; post-fix, 27/27 clean, and every partition resumes consuming after the downgrade revoke.
  • kfake suite -race -skip ETL: pass, zero races.
  • 85 varied -race runs of pure cooperative-sticky churn confirm the invariant holds without a downgrade (the flip is the only feeder found; two independent full traces of the load/fetch ownership machinery agreed).

#1271 and the "partition fetched while offsets concurrently fetched" oddity in #1222 share this signature and were both mixed-config-capable groups; this plausibly retires those as well.

Closes #1365

twmb added 2 commits July 7, 2026 23:58
…ative to eager

A group whose members are configured with both cooperative and eager
balancers (the standard migration config) runs cooperatively until a
member supporting only an eager protocol joins; the group's protocol
vote is then forced to eager. We previously followed the flip with only
a warning. The prior cooperative session revoked nothing at its end,
while the eager session's diffAssigned returns the entire assignment as
newly added: fetchOffsets then assigned offsets for partitions that
were still actively fetching, giving each cursor two owners (the
in-flight fetch and the new list/epoch load). The load's completion
raced concurrent polls' cursor updates - a data race and a potentially
torn cursor offset.

The Java client pins each member's rebalance protocol at construction
from its own assignor list, so a mixed-list Java member revokes
everything on every rejoin and can never flip mid-life. We follow the
group-selected protocol per generation - cooperative rebalancing
whenever the whole group supports it - so we must handle the flip: on a
downgrade, revoke all partitions before consuming the new assignment
(invalidate everything, let the user commit in onRevoked, clear commit
state), the state an eager session expects to begin from.

A final autocommit between the downgrade sync and the revoke can land
after another member fetched offsets; that yields duplicates only and
cannot be closed client-side, since we only learn the selected protocol
from the join response.

Closes #1365
Regression for #1365: one member advertises [cooperative-sticky,
sticky] and consumes hot while a second joins advertising only
[sticky], forcing the group's protocol vote to sticky. Pre-fix this
raced under -race in most runs; post-fix it is clean and every
partition resumes consuming after the downgrade revoke.
@twmb twmb force-pushed the kgo-cooperative-downgrade branch from 92c5e7f to bac25da Compare July 8, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data race on cursor offset between PollFetches and list/epoch load results during rebalance

1 participant