From ee01fffc077d12bfee07122f73961ef8c8600438 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Tue, 25 Jul 2023 12:52:47 -0700 Subject: [PATCH 1/3] gh-106939: document ShareableList nul-strip quirk. --- Doc/library/multiprocessing.shared_memory.rst | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/Doc/library/multiprocessing.shared_memory.rst b/Doc/library/multiprocessing.shared_memory.rst index 76046b34610abe..1abb368a41c106 100644 --- a/Doc/library/multiprocessing.shared_memory.rst +++ b/Doc/library/multiprocessing.shared_memory.rst @@ -275,6 +275,35 @@ shared memory blocks created using that manager are all released when the existing ``ShareableList``, specify its shared memory block's unique name while leaving ``sequence`` set to ``None``. + .. note:: + + A known issue exists for :class:`bytes` and :class:`str` values. + If they end with ``\x00`` nul bytes or characters, those may be + *silently stripped* when fetching them by index from the + :class:`ShareableList`. This ``.rstrip(b'\x00')`` behavior is + considered a bug and may go away in the future. See :gh:`106939`. + + For applications where rstripping of trailing nulls is a problem, + work around it by always unconditionally appending an extra non-0 + byte to the end of such values when storing and unconditionally + removing it when fetching: + + .. doctest:: + + >>> from multiprocessing import shared_memory + >>> nul_bug_demo = shared_memory.ShareableList(['?\x00', b'\x03\x02\x01\x00\x00\x00']) + >>> nul_bug_demo[0] + '?' + >>> nul_bug_demo[1] + b'\x03\x02\x01' + >>> nul_bug_demo.shm.shutdown() + >>> padded = shared_memory.ShareableList(['?\x00\x07', b'\x03\x02\x01\x00\x00\x00\x07']) + >>> padded[0][:-1] + '?\x00' + >>> padded[1][:-1] + b'\x03\x02\x01\x00\x00\x00' + >>> padded.shm.shutdown() + .. method:: count(value) Returns the number of occurrences of ``value``. From 1013e304ab368f3486e8d123334cd45e19f9c55b Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Tue, 25 Jul 2023 13:09:52 -0700 Subject: [PATCH 2/3] Mention the `int` size constraint. --- Doc/library/multiprocessing.shared_memory.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/Doc/library/multiprocessing.shared_memory.rst b/Doc/library/multiprocessing.shared_memory.rst index 1abb368a41c106..f7dfc3cab29cfd 100644 --- a/Doc/library/multiprocessing.shared_memory.rst +++ b/Doc/library/multiprocessing.shared_memory.rst @@ -255,16 +255,17 @@ shared memory blocks created using that manager are all released when the :keyword:`with` statement's code block finishes execution. -.. class:: ShareableList(sequence=None, *, name=None) +.. class:: ShareableList(sequence=None, \*, name=None) Provides a mutable list-like object where all values stored within are stored in a shared memory block. This constrains storable values to - only the ``int``, ``float``, ``bool``, ``str`` (less than 10M bytes each), - ``bytes`` (less than 10M bytes each), and ``None`` built-in data types. - It also notably differs from the built-in ``list`` type in that these - lists can not change their overall length (i.e. no append, insert, etc.) - and do not support the dynamic creation of new :class:`ShareableList` - instances via slicing. + only the ``int`` (signed 64-bit), ``float``, ``bool``, ``str`` (less + than 10M bytes each when encoded as utf-8), ``bytes`` (less than 10M + bytes each), and ``None`` built-in data types. It also notably + differs from the built-in ``list`` type in that these lists can not + change their overall length (i.e. no append, insert, etc.) and do not + support the dynamic creation of new :class:`ShareableList` instances + via slicing. *sequence* is used in populating a new ``ShareableList`` full of values. Set to ``None`` to instead attach to an already existing From 59dea5a7075463867ad80e2aa279a669ef12e84a Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith [Google LLC]" Date: Tue, 25 Jul 2023 13:16:14 -0700 Subject: [PATCH 3/3] fix the doctest --- Doc/library/multiprocessing.shared_memory.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/multiprocessing.shared_memory.rst b/Doc/library/multiprocessing.shared_memory.rst index f7dfc3cab29cfd..f453e6403d932d 100644 --- a/Doc/library/multiprocessing.shared_memory.rst +++ b/Doc/library/multiprocessing.shared_memory.rst @@ -297,13 +297,13 @@ shared memory blocks created using that manager are all released when the '?' >>> nul_bug_demo[1] b'\x03\x02\x01' - >>> nul_bug_demo.shm.shutdown() + >>> nul_bug_demo.shm.unlink() >>> padded = shared_memory.ShareableList(['?\x00\x07', b'\x03\x02\x01\x00\x00\x00\x07']) >>> padded[0][:-1] '?\x00' >>> padded[1][:-1] b'\x03\x02\x01\x00\x00\x00' - >>> padded.shm.shutdown() + >>> padded.shm.unlink() .. method:: count(value)