Crash on deallocator of type 'time.struct_time' during finalizing for free-threaded Python #122527

XuehaiPan · 2024-07-31T17:29:15Z

Crash report

What happened?

A simple standalone file:

# test1.py

import time

obj = time.struct_time(range(1, 10))

$ pyenv install 3.14t-dev --debug
$ pyenv local 3.14t-dev-debug
$ python3 -VV
Python 3.14.0a0 experimental free-threading build (heads/main:bd3d31f, Aug  1 2024, 01:22:33) [Clang 15.0.0 (clang-1500.3.9.4)]
$ python3 test1.py
Fatal Python error: _Py_Dealloc: Deallocator of type 'time.struct_time' raised an exception
Python runtime state: finalizing (tstate=0x00000001018ca548)
Exception ignored in the internal traceback machinery:
ImportError: sys.meta_path is None, Python is likely shutting down
TypeError: Missed attribute 'n_fields' of type time.struct_time

Current thread 0x00000001fff70c00 (most recent call first):
  Garbage-collecting
  <no Python frame>
[1]    89377 abort      python3 test1.py
$ python3 -c 'import test1'
Fatal Python error: _Py_Dealloc: Deallocator of type 'time.struct_time' raised an exception
Python runtime state: finalizing (tstate=0x00000001040a2548)
Exception ignored in the internal traceback machinery:
ImportError: sys.meta_path is None, Python is likely shutting down
TypeError: Missed attribute 'n_fields' of type time.struct_time

Current thread 0x00000001fff70c00 (most recent call first):
  Garbage-collecting
  <no Python frame>
[1]    89737 abort      python3 -c 'import test1'

If I remove the assignment for obj, the program can exit successfully.

# test2.py

import time

time.struct_time(range(1, 10))

$ python3 test2.py           # no output
$ python3 -c 'import test2'  # no output

CPython versions tested on:

CPython main branch, CPython 3.13t, CPython 3.14t

Operating systems tested on:

Linux, macOS

Output from running 'python -VV' on the command line:

Python 3.14.0a0 experimental free-threading build (heads/main:bd3d31f, Aug 1 2024, 01:22:33) [Clang 15.0.0 (clang-1500.3.9.4)]

Linked PRs

The text was updated successfully, but these errors were encountered:

Eclips4 · 2024-07-31T17:34:04Z

Thanks for the report!
Confirmed on current main.

colesbury · 2024-07-31T20:55:21Z

Bisected to:

4ad8f09 (gh-117376: Partial implementation of deferred reference counting #117696)

(oops)

colesbury · 2024-07-31T21:27:38Z

The problem is in structseq_dealloc, the REAL_SIZE() call fails:

cpython/Objects/structseq.c

Lines 116 to 131 in 8844197

    
           static void 
        
           structseq_dealloc(PyStructSequence *obj) 
        
           { 
        
               Py_ssize_t i, size; 
        
               PyObject_GC_UnTrack(obj); 
        
               PyTypeObject *tp = Py_TYPE(obj); 
        
               size = REAL_SIZE(obj); 
        
               for (i = 0; i < size; ++i) { 
        
                   Py_XDECREF(obj->ob_item[i]); 
        
               } 
        
               PyObject_GC_Del(obj); 
        
               if (_PyType_HasFeature(tp, Py_TPFLAGS_HEAPTYPE)) { 
        
                   Py_DECREF(tp); 
        
               } 
        
           }

REAL_SIZE is a macro that expands to:

PyDict_GetItemWithError(_PyType_GetDict(Py_TYPE(self)), _Py_ID(n_fields))

The GC is clearing the type's dictionary (i.e., _PyType_GetDict) before the structseq instances is freed, so the REAL_SIZE call fails. We're also not handling that failure, which leads to a crash.

The default build works because the GC happens to free the structseq instance first before calling tp_clear() on the type's dictionary, but I don't think the GC guarantees any ordering of tp_clear() calls.

colesbury · 2024-07-31T21:40:13Z

You can trigger a similar crash in the default build with some weirder code:

import time
obj = time.struct_time(range(1, 10))
type(obj).obj = obj

The `PyStructSequence` destructor would crash if it was deallocated after its type's dictionary was cleared by the GC, because it couldn't compute the "real size" of the instance. This could occur with relatively straightforward code in the free-threaded build or with a reference cycle involving the type in the default build, due to differing orders in which `tp_clear()` was called. Account for the non-sequence fields in `tp_basicsize` and use that, along with `Py_SIZE()`, to compute the "real" size of a `PyStructSequence` in the dealloc function. This avoids the accesses to the type's dictionary during dealloc, which were unsafe.

…ythonGH-122577) The `PyStructSequence` destructor would crash if it was deallocated after its type's dictionary was cleared by the GC, because it couldn't compute the "real size" of the instance. This could occur with relatively straightforward code in the free-threaded build or with a reference cycle involving the type in the default build, due to differing orders in which `tp_clear()` was called. Account for the non-sequence fields in `tp_basicsize` and use that, along with `Py_SIZE()`, to compute the "real" size of a `PyStructSequence` in the dealloc function. This avoids the accesses to the type's dictionary during dealloc, which were unsafe. (cherry picked from commit 4b63cd1) Co-authored-by: Sam Gross <[email protected]>

…ythonGH-122577) The `PyStructSequence` destructor would crash if it was deallocated after its type's dictionary was cleared by the GC, because it couldn't compute the "real size" of the instance. This could occur with relatively straightforward code in the free-threaded build or with a reference cycle involving the type in the default build, due to differing orders in which `tp_clear()` was called. Account for the non-sequence fields in `tp_basicsize` and use that, along with `Py_SIZE()`, to compute the "real" size of a `PyStructSequence` in the dealloc function. This avoids the accesses to the type's dictionary during dealloc, which were unsafe.

…H-122577) (#122625) gh-122527: Fix a crash on deallocation of `PyStructSequence` (GH-122577) The `PyStructSequence` destructor would crash if it was deallocated after its type's dictionary was cleared by the GC, because it couldn't compute the "real size" of the instance. This could occur with relatively straightforward code in the free-threaded build or with a reference cycle involving the type in the default build, due to differing orders in which `tp_clear()` was called. Account for the non-sequence fields in `tp_basicsize` and use that, along with `Py_SIZE()`, to compute the "real" size of a `PyStructSequence` in the dealloc function. This avoids the accesses to the type's dictionary during dealloc, which were unsafe. (cherry picked from commit 4b63cd1) Co-authored-by: Sam Gross <[email protected]>

…H-122577) (#122626) The `PyStructSequence` destructor would crash if it was deallocated after its type's dictionary was cleared by the GC, because it couldn't compute the "real size" of the instance. This could occur with relatively straightforward code in the free-threaded build or with a reference cycle involving the type in the default build, due to differing orders in which `tp_clear()` was called. Account for the non-sequence fields in `tp_basicsize` and use that, along with `Py_SIZE()`, to compute the "real" size of a `PyStructSequence` in the dealloc function. This avoids the accesses to the type's dictionary during dealloc, which were unsafe. (cherry picked from commit 4b63cd1)

XuehaiPan added the type-crash A hard crash of the interpreter, possibly with a core dump label Jul 31, 2024

Eclips4 added the topic-free-threading label Jul 31, 2024

colesbury self-assigned this Jul 31, 2024

colesbury added 3.13 bugs and security fixes 3.14 bugs and security fixes labels Jul 31, 2024

bedevere-app bot mentioned this issue Aug 1, 2024

gh-122527: Fix a crash on deallocation of PyStructSequence #122577

Merged

This was referenced Aug 2, 2024

[3.13] gh-122527: Fix a crash on deallocation of PyStructSequence (GH-122577) #122625

Merged

[3.12] gh-122527: Fix a crash on deallocation of PyStructSequence (GH-122577) #122626

Merged

XuehaiPan mentioned this issue Sep 1, 2024

deps(python): enable Python 3.13t support metaopt/optree#137

Merged

12 tasks

ambv closed this as completed Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on deallocator of type 'time.struct_time' during finalizing for free-threaded Python #122527

Crash on deallocator of type 'time.struct_time' during finalizing for free-threaded Python #122527

XuehaiPan commented Jul 31, 2024 •

edited by bedevere-app bot

Loading

Eclips4 commented Jul 31, 2024

colesbury commented Jul 31, 2024

colesbury commented Jul 31, 2024

colesbury commented Jul 31, 2024 •

edited

Loading

Crash on deallocator of type 'time.struct_time' during finalizing for free-threaded Python #122527

Crash on deallocator of type 'time.struct_time' during finalizing for free-threaded Python #122527

Comments

XuehaiPan commented Jul 31, 2024 • edited by bedevere-app bot Loading

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Eclips4 commented Jul 31, 2024

colesbury commented Jul 31, 2024

colesbury commented Jul 31, 2024

colesbury commented Jul 31, 2024 • edited Loading

XuehaiPan commented Jul 31, 2024 •

edited by bedevere-app bot

Loading

colesbury commented Jul 31, 2024 •

edited

Loading