Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-125828 : Fix 'get_value' missing for [Bounded]Semaphore on multiprocessing MacOSX - C workaround #130913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

YvesDup
Copy link
Contributor

@YvesDup YvesDup commented Mar 6, 2025

Following #125829 discussions and after a long deeping in the _multiprocessing.semaphore.c C file, I suggest a workaround based on a shared memory that contains an array to store all the internal semaphore counters.

At the moment, all the tests are passing, but I'm having 2 side problems, the second of which is really embarrassing:

  1. How to calculate the size of the array stored in the shared memory, knowing that:
    1. I sized the array from the value of sysconf(SC_SEM_NSEMS_MAX) which is set to 87381 on a mac OSX 15. This is very high.
    2. But I can't create more than about 4900 semaphores regardless of the number of python instances. I get an OSError(28, 'No space left on device'). What is the right value ?
  2. When and how to call the system function shm_unlink to properly close the shared memory (First I called shm_open)?

Regarding this last point, I wonder if only the process that opened the shared memory should call this function? Or should all processes that called shm_open call shm_unlink? Unfortunately, the Posix documentation isn't precise.

I worked on the count of opened/closed processes, with the aim of calling shm_unlink when the last process has closed, without success. I also had a look to the posixshmem module, without success too.
Could the resource_tracker and/or the use of the Finalize class of the multiprocessing.util module help me to fix this problem ? Is it possible to use these 2 Python libs from C API ?

This memory management difficulty make me think that this choice may not be appropriate. I am a bit stuck.
Thank you for your feedback and ideas.

@YvesDup YvesDup marked this pull request as draft April 15, 2025 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant