Codestin Search App

J-HowHuang · 2026-03-03T04:25:30Z

Description

For tables using FD_AWARE_INSTANCE_PARTITION_SELECTOR as their partition selector in instance assignment config, it's likely to fail rebalance when minimizeDataMovement=true if the instances didn't change in all pools. This will result in throwing an exception here java.util.NoSuchElementException since the map is empty

pinot/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/FDAwareInstancePartitionSelector.java

Lines 250 to 255 in 1ebc1af

    
           CandidateQueue(Map<Integer, LinkedHashSet<String>> faultDomainToCandidateInstancesMap) { 
        
             _map = new TreeMap<>(); 
        
             faultDomainToCandidateInstancesMap.entrySet().stream().filter(kv -> !kv.getValue().isEmpty()) 
        
                 .forEach(kv -> _map.put(kv.getKey(), new LinkedList<>(kv.getValue()))); 
        
             _iter = _map.firstKey(); 
        
           }

Reproduce

Run quickstart offline, for table airlineStats_OFFLINE, remove its tierConfigs and add the following instanceAssignmentConfigMap

"instanceAssignmentConfigMap": {
      "OFFLINE": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_OFFLINE",
          "poolBased": false,
          "numPools": 1
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 1,
          "numInstancesPerReplicaGroup": 1,
          "numPartitions": 0,
          "numInstancesPerPartition": 0,
          "minimizeDataMovement": false
        },
        "partitionSelector": "FD_AWARE_INSTANCE_PARTITION_SELECTOR",
        "minimizeDataMovement": false
      }
    }

Run rebalance with minimize data movement enabled.
Results in

{
  "jobId": "b6c56b27-450a-49c0-ac69-16835ed47fff",
  "status": "FAILED",
  "description": "Caught exception while fetching/calculating instance partitions: java.util.NoSuchElementException"
}

Controller log:

2026/03/03 12:23:37.194 INFO [InstanceTagPoolSelector] [grizzly-http-server-2] Selecting 1 instances for table: airlineStats_OFFLINE
2026/03/03 12:23:37.194 INFO [InstanceAssignmentDriver] [grizzly-http-server-2] No instance constraint is configured, using default hash-based-rotate instance constraint
2026/03/03 12:23:37.194 INFO [HashBasedRotateInstanceConstraintApplier] [grizzly-http-server-2] Rotating instances for table: airlineStats_OFFLINE with hash: 802879867
2026/03/03 12:23:37.194 INFO [FDAwareInstancePartitionSelector] [grizzly-http-server-2] Assigning 1 replica groups to 1 fault domains
2026/03/03 12:23:37.194 INFO [FDAwareInstancePartitionSelector] [grizzly-http-server-2] Warning, normalizing isn't finished yet
2026/03/03 12:23:37.195 WARN [TableRebalancer-airlineStats_OFFLINE-2b912476-fb58-4b47-907a-7db8f87d85c1] [grizzly-http-server-2] Caught exception while fetching/calculating instance partitions, aborting the rebalance
java.util.NoSuchElementException
	at java.base/java.util.TreeMap.key(TreeMap.java:1602)
	at java.base/java.util.TreeMap.firstKey(TreeMap.java:291)
	at org.apache.pinot.controller.helix.core.assignment.instance.FDAwareInstancePartitionSelector$CandidateQueue.<init>(FDAwareInstancePartitionSelector.java:255)
	at org.apache.pinot.controller.helix.core.assignment.instance.FDAwareInstancePartitionSelector$ReplicaGroupBasedAssignmentState.fill(FDAwareInstancePartitionSelector.java:395)
	at org.apache.pinot.controller.helix.core.assignment.instance.FDAwareInstancePartitionSelector.selectInstances(FDAwareInstancePartitionSelector.java:195)
	at org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver.getInstancePartitions(InstanceAssignmentDriver.java:154)
	at org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver.getInstancePartitions(InstanceAssignmentDriver.java:126)
	at org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver.assignInstances(InstanceAssignmentDriver.java:69)
	at org.apache.pinot.controller.helix.core.rebalance.TableRebalancer.getInstancePartitions(TableRebalancer.java:1274)

Change

Add a check before FDAwareInstancePartitionSelector$ReplicaGroupBasedAssignmentState.fill to see if there's any instance to fill, otherwise skip.

codecov-commenter · 2026-03-03T05:20:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.24%. Comparing base (7f70124) to head (56a7113).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17799      +/-   ##
============================================
+ Coverage     63.21%   63.24%   +0.03%     
- Complexity     1454     1456       +2     
============================================
  Files          3183     3186       +3     
  Lines        191500   191615     +115     
  Branches      29289    29315      +26     
============================================
+ Hits         121048   121183     +135     
+ Misses        60986    60947      -39     
- Partials       9466     9485      +19

Flag	Coverage Δ
custom-integration1	`100.00% <ø> (ø)`
integration	`100.00% <ø> (ø)`
integration1	`100.00% <ø> (ø)`
integration2	`0.00% <ø> (ø)`
java-11	`63.22% <100.00%> (+0.03%)`	⬆️
java-21	`63.22% <100.00%> (+0.03%)`	⬆️
temurin	`63.24% <100.00%> (+0.03%)`	⬆️
unittests	`63.23% <100.00%> (+0.03%)`	⬆️
unittests1	`55.61% <ø> (+0.03%)`	⬆️
unittests2	`34.14% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

yashmayya

LGTM, thanks for the fix @J-HowHuang! Is it possible to add a small test for this case?

Also, IIUC, does this mean that all steady state rebalances (no instance additions / removals) with minimizeDataMovement currently fail when the instance selector is FDAwareInstancePartitionSelector?

yashmayya · 2026-03-03T18:47:49Z

...apache/pinot/controller/helix/core/assignment/instance/FDAwareInstancePartitionSelector.java

    public void fill(Map<Integer, LinkedHashSet<String>> faultDomainToCandidateInstancesMap) {
+      // skip filling if there is no candidate instance, which can happen when minimize data movement is enabled and
+      // no new instances are added to any pool
+      if (faultDomainToCandidateInstancesMap.values().stream().allMatch(HashSet::isEmpty)) {


nit: Set::isEmpty or Collection::isEmpty would be more idiomatic rather than using HashSet::isEmpty for a LinkedHashSet.

yashmayya

LGTM, thanks for the fix @J-HowHuang! Is it possible to add a small test for this case?

Also, IIUC, does this mean that all steady state rebalances (no instance additions / removals) with minimizeDataMovement currently fail when the instance selector is FDAwareInstancePartitionSelector?

J-HowHuang · 2026-03-04T03:01:45Z

@yashmayya Added the test and verified that it would fail without the fix.

yashmayya · 2026-03-04T05:32:25Z

.../java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java

+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements.  See the NOTICE
+ * file distributed with this work for additional information regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
+ * License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations under the License.


Should be reverted

check before filling remaining instances

34c1366

J-HowHuang added bugfix segment-rebalance labels Mar 3, 2026

J-HowHuang requested a review from yashmayya March 3, 2026 04:28

move the check

1ebc1af

yashmayya approved these changes Mar 3, 2026

View reviewed changes

add unit test

56a7113

yashmayya reviewed Mar 4, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug fix] Fault-Domain-Aware Instance Assignment failing rebalance with minimize data movement#17799

[Bug fix] Fault-Domain-Aware Instance Assignment failing rebalance with minimize data movement#17799
J-HowHuang wants to merge 3 commits intoapache:masterfrom
J-HowHuang:fix-FD-aware-selector

J-HowHuang commented Mar 3, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Mar 3, 2026 •

edited

Loading

Uh oh!

yashmayya left a comment

Uh oh!

yashmayya Mar 3, 2026

Uh oh!

J-HowHuang Mar 4, 2026

Uh oh!

yashmayya left a comment

Uh oh!

J-HowHuang commented Mar 4, 2026

Uh oh!

yashmayya Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	CandidateQueue(Map<Integer, LinkedHashSet<String>> faultDomainToCandidateInstancesMap) {
	_map = new TreeMap<>();
	faultDomainToCandidateInstancesMap.entrySet().stream().filter(kv -> !kv.getValue().isEmpty())
	.forEach(kv -> _map.put(kv.getKey(), new LinkedList<>(kv.getValue())));
	_iter = _map.firstKey();
	}

Conversation

J-HowHuang commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reproduce

Change

Uh oh!

codecov-commenter commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yashmayya left a comment

Choose a reason for hiding this comment

Uh oh!

yashmayya Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

J-HowHuang Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

yashmayya left a comment

Choose a reason for hiding this comment

Uh oh!

J-HowHuang commented Mar 4, 2026

Uh oh!

yashmayya Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

J-HowHuang commented Mar 3, 2026 •

edited

Loading

codecov-commenter commented Mar 3, 2026 •

edited

Loading