FIX generator unordered lock - #1789

Codecov Report

❌ Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.18%. Comparing base (6e9a70e) to head (241bae8).
⚠️ Report is 26 commits behind head on main.

@@            Coverage Diff             @@
##             main    #1789      +/-   ##
==========================================
- Coverage   94.21%   94.18%   -0.04%     
==========================================
  Files          46       46              
  Lines        7922     7926       +4     
==========================================
+ Hits         7464     7465       +1     
- Misses        458      461       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@Nanored4498 thank you for the PR

I suggest not using a lock there. Instead, is there a way to select an item from the set, without removing said item, and without this iter trick ?

Maybe by pop'ing the item and putting it back afterward ? or having an additional attribute to joblib.Parallel ?

Reading through the retrieve function I think poping (and controlling for KeyError instead of having the None default) and adding back afterward will work.

Reading through the retrieve function I think poping (and controlling for KeyError instead of having the None default) and adding back afterward will work.

I'm not sufficiently used to python ^^ It would work because of the GIL, right? pop and add are both atomic operations, and only one thread could execute one of these methods at the same time?

TY for the update @Nanored4498

Yes pop and add are atomics, but it's not required for the logic to be sound here. This bug is related to concurrency, but is less caused by concurrency, than by the wrong usage of iter. The issue is that implementing selection of an arbitrary element of a set with next(iter(... was in insight wrong, I thought it was a cool syntax shortcut at the time, but the implementation of iter itself makes it subject to an exception triggered by a race condition.

I'd be +1 on this change, could it ship in a patch release ? The bug can really trigger at any time for any user workload :-/

@Nanored4498 could you add an entry in the changelog?

The windows failure seems to be unrelated, so we should merge this one.

@Nanored4498 could you add an entry in the changelog?

This is done ;)