Improve documentation on multiprocessing synchronization-between-processes #116526

HCharlie · 2024-03-08T23:35:06Z

Documentation

https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes
(A clear and concise description of the issue.)

when running the provided example locally, unlike other examples on this page which can print valid stdout, the code in this section will only print out the error below, which is caused by not providing the Process.join() call at the end.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

An example patch could be

from multiprocessing import Process, Lock

   def f(l, i):
       l.acquire()
       try:
           print('hello world', i)
       finally:
           l.release()

   if __name__ == '__main__':
       lock = Lock()

       processes = []

       for num in range(10):
           p = Process(target=f, args=(lock, num))
           processes.append(p)
           p.start()
   
       for p in processes:
           p.join()

Linked PRs

GH-116526 patch documentation Synchronization between processes example #116527

gaogaotiantian · 2024-03-09T22:42:47Z

To be honest this is a pretty bad example to begin with - because you won't notice any difference if you simply remove the lock so it's pretty confusing when users try it out. The stdout is buffered in most environments and won't flush until seeing a newline. We should probably change this to something more obvious - multiple prints maybe (building a triangle?).

The exception mentioned in this issue can be reproduced with spawn, but might be hard to repro with fork as fork is too fast.

HCharlie · 2024-03-10T14:22:51Z

Hey @gaogaotiantian , thanks for your reply. I think your point makes sense, and it still confuses me when reading it as well. Either with or without the lock, the example doesn't seem to emphasize what difference this lock brings. Tbh, my original intent was to just complete the example to remove some exceptions, now I think maybe a new example is needed.

HCharlie · 2024-03-10T14:56:04Z

one quick example I could think of is as below, we increment a value 3 times in a process.

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, lock, process_id):
    lock.acquire()
    for _ in range(3):
        counter.value += 1
        print(f"Counter value: {counter.value} incremented by the {process_id} process")
    lock.release()

if __name__ == "__main__":
    # Shared counter value
    counter = multiprocessing.Value('i', 0)
    
    # Creating a Lock
    lock = multiprocessing.Lock()

    # Creating processes
    processes = []
    for j in range(4):  # Creating 4 processes
        p = multiprocessing.Process(target=increment_counter, args=(counter, lock, j))
        processes.append(p)
        p.start()

    # Waiting for all processes to finish
    for p in processes:
        p.join()

    # Output the final value of the counter
    print("Final counter value:", counter.value)

with the lock, the value will consecutively be incremented by the same process, and the example will output something like this

Counter value: 1 incremented by the 1 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 1 process
Counter value: 4 incremented by the 0 process
Counter value: 5 incremented by the 0 process
Counter value: 6 incremented by the 0 process
Counter value: 7 incremented by the 2 process
Counter value: 8 incremented by the 2 process
Counter value: 9 incremented by the 2 process
Counter value: 10 incremented by the 3 process
Counter value: 11 incremented by the 3 process
Counter value: 12 incremented by the 3 process
Final counter value: 12

if we remove the lock-related code,

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, process_id):

    for _ in range(3):
        counter.value += 1
        print(f"Counter value: {counter.value} incremented by the {process_id} process")


if __name__ == "__main__":
    # Shared counter value
    counter = multiprocessing.Value('i', 0)

    # Creating processes
    processes = []
    for j in range(4):  # Creating 4 processes
        p = multiprocessing.Process(target=increment_counter, args=(counter, j))
        processes.append(p)
        p.start()

    # Waiting for all processes to finish
    for p in processes:
        p.join()

    # Output the final value of the counter
    print("Final counter value:", counter.value)

it will randomly print out some duplicates like below, eventually not even reaching 12.

Counter value: 1 incremented by the 3 process
Counter value: 1 incremented by the 2 process
Counter value: 1 incremented by the 0 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 3 process
Counter value: 4 incremented by the 2 process
Counter value: 4 incremented by the 0 process
Counter value: 4 incremented by the 1 process
Counter value: 5 incremented by the 3 process
Counter value: 6 incremented by the 2 process
Counter value: 6 incremented by the 0 process
Counter value: 6 incremented by the 1 process
Final counter value: 6

wdyt?

HCharlie added the docs Documentation in the Doc dir label Mar 8, 2024

HCharlie pushed a commit to HCharlie/cpython that referenced this issue Mar 8, 2024

pythonGH-116526 patch Synchronization between processes example

7f62921

HCharlie mentioned this issue Mar 8, 2024

GH-116526 patch documentation Synchronization between processes example #116527

Open

HCharlie added a commit to HCharlie/cpython that referenced this issue Mar 8, 2024

pythonGH-116526 patch Synchronization between processes example

12981a5

HCharlie added a commit to HCharlie/cpython that referenced this issue Mar 9, 2024

pythonGH-116526 patch Synchronization between processes example

31fe118

vstinner added the topic-multiprocessing label Mar 26, 2024

Moosems mentioned this issue Aug 21, 2024

Creating multiprocessing.Queues operates strangely with Process's created in classes #123208

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve documentation on multiprocessing synchronization-between-processes #116526

Improve documentation on multiprocessing synchronization-between-processes #116526

HCharlie commented Mar 8, 2024 •

edited by hugovk

Loading

gaogaotiantian commented Mar 9, 2024

HCharlie commented Mar 10, 2024 •

edited

Loading

HCharlie commented Mar 10, 2024 •

edited by hugovk

Loading

Improve documentation on multiprocessing synchronization-between-processes #116526

Improve documentation on multiprocessing synchronization-between-processes #116526

Comments

HCharlie commented Mar 8, 2024 • edited by hugovk Loading

Documentation

Linked PRs

gaogaotiantian commented Mar 9, 2024

HCharlie commented Mar 10, 2024 • edited Loading

HCharlie commented Mar 10, 2024 • edited by hugovk Loading

HCharlie commented Mar 8, 2024 •

edited by hugovk

Loading

HCharlie commented Mar 10, 2024 •

edited

Loading

HCharlie commented Mar 10, 2024 •

edited by hugovk

Loading