SF bug 1185883: PyObject_Realloc can't safely take over a block currently

tim-one · tim-one · commit ecc6e6a54ed4 · 2005-07-10T22:30:55.000Z
managed by C, because it's possible for the block to be smaller than the
new requested size, and at the end of allocated VM.  Trying to copy over
nbytes bytes to a Python small-object block can segfault then, and there's
no portable way to avoid this (we would have to know how many bytes
starting at p are addressable, and std C has no means to determine that).

Bugfix candidate.  Should be backported to 2.4, but I'm out of time.
diff --git a/Misc/NEWS b/Misc/NEWS
@@ -12,6 +12,15 @@ What's New in Python 2.5 alpha 1?
 Core and builtins
 -----------------
 
+- SF bug #1185883:  Python's small-object memory allocator took over
+  a block managed by the platform C library whenever a realloc specified
+  a small new size.  However, there's no portable way to know then how
+  much of the address space following the pointer is valid, so no
+  portable way to copy data from the C-managed block into Python's
+  small-object space without risking a memory fault.  Python's small-object
+  realloc now leaves such blocks under the control of the platform C
+  realloc.
+
 - SF bug #1232517: An overflow error was not detected properly when
   attempting to convert a large float to an int in os.utime().
 
@@ -59,7 +68,7 @@ Core and builtins
 - Bug #1165306: instancemethod_new allowed the creation of a method
   with im_class == im_self == NULL, which caused a crash when called.
 
-- Move exception finalisation later in the shutdown process - this 
+- Move exception finalisation later in the shutdown process - this
   fixes the crash seen in bug #1165761
 
 - Added two new builtins, any() and all().
@@ -74,15 +83,15 @@ Core and builtins
 - Bug #1155938: new style classes did not check that __init__() was
   returning None.
 
-- Patch #802188: Report characters after line continuation character 
+- Patch #802188: Report characters after line continuation character
   ('\') with a specific error message.
 
 - Bug #723201: Raise a TypeError for passing bad objects to 'L' format.
 
 - Bug #1124295: the __name__ attribute of file objects was
   inadvertently made inaccessible in restricted mode.
 
-- Bug #1074011: closing sys.std{out,err} now causes a flush() and 
+- Bug #1074011: closing sys.std{out,err} now causes a flush() and
   an ferror() call.
 
 - min() and max() now support key= arguments with the same meaning as in
@@ -103,7 +112,7 @@ Core and builtins
 Extension Modules
 -----------------
 
-- Bug #1234979: For the argument of thread.Lock.acquire, the Windows 
+- Bug #1234979: For the argument of thread.Lock.acquire, the Windows
   implemented treated all integer values except 1 as false.
 
 - Bug #1194181: bz2.BZ2File didn't handle mode 'U' correctly.
@@ -128,7 +137,7 @@ Extension Modules
 - Patches #925152, #1118602: Avoid reading after the end of the buffer
   in pyexpat.GetInputContext.
 
-- Patches #749830, #1144555: allow UNIX mmap size to default to current 
+- Patches #749830, #1144555: allow UNIX mmap size to default to current
   file size.
 
 - Added functional.partial().  See PEP309.
@@ -201,7 +210,7 @@ Library
 - Bug #1163325:  Decimal infinities failed to hash.  Attempting to
   hash a NaN raised an InvalidOperation instead of a TypeError.
 
-- Patch #918101: Add tarfile open mode r|* for auto-detection of the 
+- Patch #918101: Add tarfile open mode r|* for auto-detection of the
   stream compression; add, for symmetry reasons, r:* as a synonym of r.
 
 - Patch #1043890: Add extractall method to tarfile.
@@ -212,7 +221,7 @@ Library
 - Patch #1103407: Properly deal with tarfile iterators when untarring
   symbolic links on Windows.
 
-- Patch #645894: Use getrusage for computing the time consumption in 
+- Patch #645894: Use getrusage for computing the time consumption in
   profile.py if available.
 
 - Patch #1046831: Use get_python_version where appropriate in sysconfig.py.
@@ -250,7 +259,7 @@ Library
 
   + Dialects are now validated by the underlying C code, better
     reflecting it's capabilities, and improving it's compliance with
-    PEP 305. 
+    PEP 305.
   + Dialect parameter parsing has been re-implemented to improve error
     reporting.
   + quotechar=None and quoting=QUOTE_NONE now work the way PEP 305
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
@@ -139,9 +139,9 @@
  * getpagesize() call or deduced from various header files. To make
  * things simpler, we assume that it is 4K, which is OK for most systems.
  * It is probably better if this is the native page size, but it doesn't
- * have to be.  In theory, if SYSTEM_PAGE_SIZE is larger than the native page 
- * size, then `POOL_ADDR(p)->arenaindex' could rarely cause a segmentation 
- * violation fault.  4K is apparently OK for all the platforms that python 
+ * have to be.  In theory, if SYSTEM_PAGE_SIZE is larger than the native page
+ * size, then `POOL_ADDR(p)->arenaindex' could rarely cause a segmentation
+ * violation fault.  4K is apparently OK for all the platforms that python
  * currently targets.
  */
 #define SYSTEM_PAGE_SIZE	(4 * 1024)
@@ -841,30 +841,26 @@ PyObject_Realloc(void *p, size_t nbytes)
 		}
 		return bp;
 	}
-	/* We're not managing this block. */
-	if (nbytes <= SMALL_REQUEST_THRESHOLD) {
-		/* Take over this block -- ask for at least one byte so
-		 * we really do take it over (PyObject_Malloc(0) goes to
-		 * the system malloc).
-		 */
-		bp = PyObject_Malloc(nbytes ? nbytes : 1);
-		if (bp != NULL) {
-			memcpy(bp, p, nbytes);
-			free(p);
-		}
-		else if (nbytes == 0) {
-			/* Meet the doc's promise that nbytes==0 will
-			 * never return a NULL pointer when p isn't NULL.
-			 */
-			bp = p;
-		}
-
-	}
-	else {
-		assert(nbytes != 0);
-		bp = realloc(p, nbytes);
-	}
-	return bp;
+	/* We're not managing this block.  If nbytes <=
+	 * SMALL_REQUEST_THRESHOLD, it's tempting to try to take over this
+	 * block.  However, if we do, we need to copy the valid data from
+	 * the C-managed block to one of our blocks, and there's no portable
+	 * way to know how much of the memory space starting at p is valid.
+	 * As bug 1185883 pointed out the hard way, it's possible that the
+	 * C-managed block is "at the end" of allocated VM space, so that
+	 * a memory fault can occur if we try to copy nbytes bytes starting
+	 * at p.  Instead we punt:  let C continue to manage this block.
+         */
+	if (nbytes)
+		return realloc(p, nbytes);
+	/* C doesn't define the result of realloc(p, 0) (it may or may not
+	 * return NULL then), but Python's docs promise that nbytes==0 never
+	 * returns NULL.  We don't pass 0 to realloc(), to avoid that endcase
+	 * to begin with.  Even then, we can't be sure that realloc() won't
+	 * return NULL.
+	 */
+	bp = realloc(p, 1);
+   	return bp ? bp : p;
 }
 
 #else	/* ! WITH_PYMALLOC */
