Thanks to visit codestin.com
Credit goes to github.com

Skip to content

LocalCluster fails to spawn requested number of workers (capped at 5 workers on Windows) #9134

@jianlinshi

Description

@jianlinshi

Description

When creating a LocalCluster with n_workers > 5, the cluster consistently only spawns exactly 5 workers, regardless of the requested number. This appears to be a hard limit on Windows systems.

Environment

  • OS: Windows
  • Python version: (output from python --version)
  • Dask version: 2025.10.0
  • Distributed version: 2025.10.0
  • Installation method: conda/miniforge

Minimal Reproducible Example

from dask.distributed import LocalCluster, Client
import time

if __name__ == '__main__':
    print('=== TESTING DASK LOCAL CLUSTER WITHOUT PLUGINS ===')
    
    for n_workers in [5, 7, 10, 20, 50]:
        print(f'\n--- Testing {n_workers} workers ---')
        
        cluster = LocalCluster(
            n_workers=n_workers,
            threads_per_worker=1,
            processes=True,
            memory_limit='500MB',
            silence_logs=True
        )
        client = Client(cluster)
        
        # Wait for workers to connect
        time.sleep(5)
        
        # Check how many workers registered
        info = client.scheduler_info()
        num_connected = len(info['workers'])
        print(f'Requested: {n_workers}, Connected: {num_connected}')
        
        client.close()
        cluster.close()
        time.sleep(2)
    
    print('\n=== TEST COMPLETE ===')

Expected Behavior

The LocalCluster should spawn the number of workers specified by the n_workers parameter. For example:

n_workers=7 should create 7 workers
n_workers=10 should create 10 workers
n_workers=50 should create 50 workers

Actual Behavior

Regardless of the n_workers parameter, exactly 5 workers are created and connected:
=== TESTING DASK LOCAL CLUSTER WITHOUT PLUGINS ===

--- Testing 5 workers ---
Requested: 5, Connected: 5

--- Testing 7 workers ---
Requested: 7, Connected: 5

--- Testing 10 workers ---
Requested: 10, Connected: 5

--- Testing 20 workers ---
Requested: 20, Connected: 5

--- Testing 50 workers ---
Requested: 50, Connected: 5

=== TEST COMPLETE ===

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs infoNeeds further information from the user

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions