PyMem_Malloc() now uses the fast pymalloc allocator

vstinner · vstinner · commit f5c4b99034fa · 2016-04-22T16:26:23.000+02:00
Issue #26249: PyMem_Malloc() allocator family now uses the pymalloc allocator rather than system malloc(). Applications calling PyMem_Malloc() without holding the GIL can now crash: use PYTHONMALLOC=debug environment variable to validate the usage of memory allocators in your application.
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
@@ -165,15 +165,17 @@ The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
 memory from the Python heap.
 
-The default memory block allocator uses the following functions:
-:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
-``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
+By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
 
 .. warning::
 
    The :term:`GIL <global interpreter lock>` must be held when using these
    functions.
 
+.. versionchanged:: 3.6
+
+   The default allocator is now pymalloc instead of system :c:func:`malloc`.
+
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
@@ -295,15 +297,32 @@ Customize Memory Allocators
 
    Enum used to identify an allocator domain. Domains:
 
-   * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
-     :c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and
-     :c:func:`PyMem_RawFree`
-   * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
-     :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free`
-   * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
-     :c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and
-     :c:func:`PyObject_Free`
+   .. c:var:: PYMEM_DOMAIN_RAW
+
+      Functions:
+
+      * :c:func:`PyMem_RawMalloc`
+      * :c:func:`PyMem_RawRealloc`
+      * :c:func:`PyMem_RawCalloc`
+      * :c:func:`PyMem_RawFree`
+
+   .. c:var:: PYMEM_DOMAIN_MEM
 
+      Functions:
+
+      * :c:func:`PyMem_Malloc`,
+      * :c:func:`PyMem_Realloc`
+      * :c:func:`PyMem_Calloc`
+      * :c:func:`PyMem_Free`
+
+   .. c:var:: PYMEM_DOMAIN_OBJ
+
+      Functions:
+
+      * :c:func:`PyObject_Malloc`
+      * :c:func:`PyObject_Realloc`
+      * :c:func:`PyObject_Calloc`
+      * :c:func:`PyObject_Free`
 
 .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
 
@@ -328,18 +347,12 @@ Customize Memory Allocators
 
 .. c:function:: void PyMem_SetupDebugHooks(void)
 
-   Setup hooks to detect bugs in the following Python memory allocator
-   functions:
-
-   - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
-     :c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree`
-   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`,
-     :c:func:`PyMem_Free`
-   - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
-     :c:func:`PyObject_Calloc`, :c:func:`PyObject_Free`
+   Setup hooks to detect bugs in the Python memory allocator functions.
 
    Newly allocated memory is filled with the byte ``0xCB``, freed memory is
-   filled with the byte ``0xDB``. Additional checks:
+   filled with the byte ``0xDB``.
+
+   Runtime checks:
 
    - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
      allocated by :c:func:`PyMem_Malloc`
@@ -377,8 +390,9 @@ to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
 with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
 :c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
 
-*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
-(ex: :c:func:`PyObject_Malloc`).
+*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex:
+:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex:
+:c:func:`PyObject_Malloc`) domains.
 
 The arena allocator uses the following functions:
 
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
@@ -628,12 +628,11 @@ conflict.
    Set the family of memory allocators used by Python:
 
    * ``malloc``: use the :c:func:`malloc` function of the C library
-     for all Python memory allocators (ex: :c:func:`PyMem_RawMalloc`,
-     :c:func:`PyMem_Malloc` and :c:func:`PyObject_Malloc`).
-   * ``pymalloc``: :c:func:`PyObject_Malloc`, :c:func:`PyObject_Calloc` and
-     :c:func:`PyObject_Realloc` use the :ref:`pymalloc allocator <pymalloc>`.
-     Other Python memory allocators (ex: :c:func:`PyMem_RawMalloc` and
-     :c:func:`PyMem_Malloc`) use :c:func:`malloc`.
+     for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`,
+     :c:data:`PYMEM_DOMAIN_OBJ`).
+   * ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for
+     :c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use
+     the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain.
 
    Install debug hooks:
 
diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst
@@ -531,5 +531,11 @@ Changes in the Python API
 Changes in the C API
 --------------------
 
+* :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator
+  <pymalloc>` rather than system :c:func:`malloc`. Applications calling
+  :c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the
+  :envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the
+  usage of memory allocators in your application. See :issue:`26249`.
+
 * :c:func:`Py_Exit` (and the main interpreter) now override the exit status
   with 120 if flushing buffered data failed.  See :issue:`5319`.
diff --git a/Misc/NEWS b/Misc/NEWS
@@ -10,6 +10,13 @@ Release date: tba
 Core and Builtins
 -----------------
 
+- Issue #26249: Memory functions of the :c:func:`PyMem_Malloc` domain
+  (:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc allocator <pymalloc>`
+  rather than system :c:func:`malloc`. Applications calling
+  :c:func:`PyMem_Malloc` without holding the GIL can now crash: use
+  ``PYTHONMALLOC=debug`` environment variable to validate the usage of memory
+  allocators in your application.
+
 - Issue #26802: Optimize function calls only using unpacking like
   ``func(*tuple)`` (no other positional argument, no keyword): avoid copying
   the tuple. Patch written by Joe Jevnik.
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
@@ -198,9 +198,9 @@ static PyMemAllocatorEx _PyMem_Raw = {
 
 static PyMemAllocatorEx _PyMem = {
 #ifdef Py_DEBUG
-    &_PyMem_Debug.mem, PYDBG_FUNCS
+    &_PyMem_Debug.obj, PYDBG_FUNCS
 #else
-    NULL, PYMEM_FUNCS
+    NULL, PYOBJ_FUNCS
 #endif
     };
 
@@ -256,7 +256,7 @@ _PyMem_SetupAllocators(const char *opt)
         PyMemAllocatorEx obj_alloc = {NULL, PYOBJ_FUNCS};
 
         PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &mem_alloc);
-        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &mem_alloc);
+        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &obj_alloc);
         PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &obj_alloc);
 
         if (strcmp(opt, "pymalloc_debug") == 0)
