Bug description
CombinedLoader(..., mode=max_size_cycle) stops immediately when one source is empty, even when another source has batches. This contradicts the mode's documented behavior of stopping after the longest iterable is exhausted and also makes len(combined_loader) disagree with the number of yielded batches.
On current master (fcef404516264798b6c582ef556eaa17e5c29bfb), the example below reports length 3 but yields no batches.
What version are you seeing the problem on?
master
How to reproduce the bug
from lightning.pytorch.utilities.combined_loader import CombinedLoader
loader = CombinedLoader([[], [1, 2, 3]], mode=max_size_cycle)
iter(loader)
print(length:, len(loader))
print(batches:, list(loader))
Actual output:
The first source raises StopIteration, so _MaxSizeCycle resets it. Because it is genuinely empty, next() raises again, and this second exception escapes from the combined iterator before the non-empty source can contribute.
Expected: the combined iterator should not silently discard all batches from the longest source. One compatible behavior would be to retain None for an empty source while yielding the non-empty source:
[([None, 1], 0, 0), ([None, 2], 1, 0), ([None, 3], 2, 0)]
Alternatively, if empty inputs are invalid for max_size_cycle, construction or iteration should reject them explicitly instead of reporting a positive length and then yielding zero batches. I have a focused local regression and a minimal implementation of the first behavior, but am holding the PR for agreement on the contract.
Environment
- Lightning: source checkout at
fcef404516264798b6c582ef556eaa17e5c29bfb
- PyTorch: 2.13.0+cpu
- Python: 3.11.9
- OS: Windows 11
More info
The existing mixed-empty fetcher test already expects an empty/non-empty pair not to be considered done for modes other than min_size, but it does not consume the iterator and therefore misses this termination path.
Prepared with assistance from OpenAI Codex; the reproduction, history/duplicate search, local regression, and proposed patch were verified locally on CPU.
Bug description
CombinedLoader(..., mode=max_size_cycle)stops immediately when one source is empty, even when another source has batches. This contradicts the mode's documented behavior of stopping after the longest iterable is exhausted and also makeslen(combined_loader)disagree with the number of yielded batches.On current
master(fcef404516264798b6c582ef556eaa17e5c29bfb), the example below reports length 3 but yields no batches.What version are you seeing the problem on?
master
How to reproduce the bug
Actual output:
The first source raises
StopIteration, so_MaxSizeCycleresets it. Because it is genuinely empty,next()raises again, and this second exception escapes from the combined iterator before the non-empty source can contribute.Expected: the combined iterator should not silently discard all batches from the longest source. One compatible behavior would be to retain
Nonefor an empty source while yielding the non-empty source:Alternatively, if empty inputs are invalid for
max_size_cycle, construction or iteration should reject them explicitly instead of reporting a positive length and then yielding zero batches. I have a focused local regression and a minimal implementation of the first behavior, but am holding the PR for agreement on the contract.Environment
fcef404516264798b6c582ef556eaa17e5c29bfbMore info
The existing mixed-empty fetcher test already expects an empty/non-empty pair not to be considered done for modes other than
min_size, but it does not consume the iterator and therefore misses this termination path.Prepared with assistance from OpenAI Codex; the reproduction, history/duplicate search, local regression, and proposed patch were verified locally on CPU.