memory issues with 2.1.0 #2284

charlienegri · 2025-04-25T10:38:41Z

charlienegri
Apr 25, 2025

we have some code which writes contourpy contours to geojson files and is run in a venv

with shapely 2.1.0 we are suddenly getting memory corruption issues for given parameter setups leading to segmentation faults with various errors like
malloc(): unaligned tcache chunk detected
corrupted double-linked list
double free or corruption (out)

these errors do not appear to happen consistently within a given parameter setup so we cannot pin them down to a specific line, but they happen in a function that manipulates a contourpy contour_generator object via several shapely functions (from_ragged_array, set_precision, unary_union, to_geojson) to write a geojson file....

the same exact setup with shapely 2.0.6 or 2.0.7 does not lead to any errors (keeping the rest of the packages on the same version, including contourpy==1.3.2 and numpy==1.26.4)

I am wondering if the optimization of from_ragged_array in 2.1.0 could play a role here and if someone has any suggestions on how to investigate this further...

for reference here is the venv pip list, shapely excluded

certifi                2025.1.31
cftime                 1.6.4.post1
click                  8.1.8
cloudpickle            3.1.1
connectorx             0.4.3
contourpy              1.3.2
cycler                 0.12.1
dask                   2023.12.1
dynaconf               3.2.10
fonttools              4.57.0
fsspec                 2025.3.2
importlib_metadata     8.6.1
importlib-resources    5.13.0
Jinja2                 3.1.6
kiwisolver             1.4.8
locket                 1.0.0
MarkupSafe             3.0.2
matplotlib             3.10.1
mysql-connector-python 8.4.0
netCDF4                1.7.2
numpy                  1.26.4
orjson                 3.10.16
packaging              25.0
pandas                 2.2.3
partd                  1.4.2
pillow                 11.2.1
pip                    25.0.1
polars                 1.27.1
pyaeralert             0.2.0
pyarrow                19.0.1
pyparsing              3.2.3
PyQt6                  6.9.0
PyQt6-Qt6              6.9.0
PyQt6_sip              13.10.0
python-dateutil        2.9.0.post0
pytz                   2025.2
PyYAML                 6.0.2
scipy                  1.15.2
setuptools             79.0.0
simplejson             3.20.1
six                    1.17.0
toml                   0.10.2
toolz                  1.0.0
tqdm                   4.67.1
typer-slim             0.15.2
typing_extensions      4.13.2
tzdata                 2025.2
xarray                 2023.12.0
zipp                   3.21.0

jorisvandenbossche · 2025-04-25T12:52:40Z

jorisvandenbossche
Apr 25, 2025
Maintainer

Such non-deterministic crashes can indeed be hard to debug. As a start, would it be possible to provide a script that (at least sometimes) gives the crash for you? So then we could see if we can reproduce the issue as well.

One thing you could try is running the code under gdb, so you can get the backtrace of the segfault, potentially pinpointing to which shapely (or GEOS) function it originates from.

1 reply

charlienegri Apr 28, 2025
Author

so far I have been trying to run with strace, which gave no further insights (to me at least)
and then gdb on our operational infrastructure (where the debug setup is limited)
what I got was this (GDB configured as "x86_64-redhat-linux-gnu")

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /modules/rhel8/user-apps/fou-modules/pyaeralert/dev/venv/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/5b/be74eb6855e0a2c043c0bec2f484bf3e9f14c0.debug
Missing separate debuginfo for /modules/rhel8/user-apps/fou-modules/pyaeralert/dev/venv/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/54/9b4c82347785459571c79239872ad31509dcf4.debug
[New Thread 0x155547bff700 (LWP 874623)]
[New Thread 0x155537dff700 (LWP 874624)]
Missing separate debuginfo for /modules/rhel8/user-apps/fou-modules/pyaeralert/dev/venv/lib/python3.10/site-packages/PIL/../pillow.libs/libXau-154567c4.so.6.0.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/82/4cf8abc4a687b96d9fb3e6011229dc0c22e172.debug
[New Thread 0x15552f50d700 (LWP 874625)]
Missing separate debuginfo for /modules/rhel8/user-apps/fou-modules/pyaeralert/dev/venv/lib/python3.10/site-packages/scipy/linalg/../../scipy.libs/libgfortran-040039e1-0352e75f.so.5.0.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/5b/be74eb6855e0a2c043c0bec2f484bf3e9f14c0.debug
Missing separate debuginfo for /modules/rhel8/user-apps/fou-modules/pyaeralert/dev/venv/lib/python3.10/site-packages/scipy/linalg/../../scipy.libs/libquadmath-96973f99-934c22de.so.0.0.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/54/9b4c82347785459571c79239872ad31509dcf4.debug
[New Thread 0x155524b34700 (LWP 874626)]
[New Thread 0x155524833700 (LWP 874628)]
[New Thread 0x155524632700 (LWP 874629)]
[New Thread 0x155524431700 (LWP 874630)]
[New Thread 0x155524230700 (LWP 874631)]
[New Thread 0x155500737700 (LWP 874632)]
[New Thread 0x155500536700 (LWP 874633)]
[New Thread 0x155500335700 (LWP 874634)]
[New Thread 0x1554f4267700 (LWP 874635)]
[New Thread 0x155517c87700 (LWP 874739)]
[New Thread 0x155511f86700 (LWP 874740)]
[New Thread 0x1555009ff700 (LWP 874741)]
[New Thread 0x1554f44e5700 (LWP 874742)]
[New Thread 0x1554e85ff700 (LWP 874743)]
[New Thread 0x1554de3fe700 (LWP 874744)]
[New Thread 0x1554de5ff700 (LWP 874745)]
[New Thread 0x1554ddbff700 (LWP 874746)]
[New Thread 0x1554dd9fe700 (LWP 874747)]
[New Thread 0x1554dd1ff700 (LWP 874748)]
[New Thread 0x1554dc9ff700 (LWP 874749)]
[New Thread 0x155491e14700 (LWP 874750)]
[New Thread 0x1554917ff700 (LWP 874751)]
[New Thread 0x155490fff700 (LWP 874752)]
[New Thread 0x1554907ff700 (LWP 874753)]
[New Thread 0x15547afff700 (LWP 874754)]
[New Thread 0x15547a9ff700 (LWP 874755)]
[New Thread 0x15547a1ff700 (LWP 874756)]
[New Thread 0x1554799ff700 (LWP 874757)]
[New Thread 0x1554791ff700 (LWP 874758)]
[New Thread 0x1554789ff700 (LWP 874759)]
[New Thread 0x15542afff700 (LWP 874760)]
[New Thread 0x15542a9ff700 (LWP 874761)]
[New Thread 0x15542a1ff700 (LWP 874762)]
[New Thread 0x1554299ff700 (LWP 874763)]
[New Thread 0x155428fff700 (LWP 874764)]
[New Thread 0x1554287ff700 (LWP 874765)]
[New Thread 0x155417fff700 (LWP 874766)]
[New Thread 0x1554179ff700 (LWP 874767)]
[New Thread 0x1554171ff700 (LWP 874768)]
[New Thread 0x1554169ff700 (LWP 874769)]
[New Thread 0x155415fff700 (LWP 874770)]
[New Thread 0x1554157ff700 (LWP 874771)]
[New Thread 0x155414fff700 (LWP 874772)]
[New Thread 0x1554147ff700 (LWP 874773)]
[New Thread 0x15539ffff700 (LWP 874774)]
[New Thread 0x15539f7ff700 (LWP 874775)]
[New Thread 0x15539efff700 (LWP 874776)]
[New Thread 0x15539e7ff700 (LWP 874777)]
[New Thread 0x15539dfff700 (LWP 874778)]
[New Thread 0x15539d7ff700 (LWP 874779)]
[New Thread 0x15539d1ff700 (LWP 874780)]
[New Thread 0x15539c7ff700 (LWP 874781)]
[New Thread 0x15537bfff700 (LWP 874782)]
[New Thread 0x15537adff700 (LWP 874783)]
[New Thread 0x15537a5ff700 (LWP 874784)]
[New Thread 0x155379dff700 (LWP 874785)]
[New Thread 0x1553795ff700 (LWP 874786)]
[New Thread 0x155378dff700 (LWP 874787)]
[New Thread 0x1553785ff700 (LWP 874788)]
[New Thread 0x15535fdff700 (LWP 874789)]
[New Thread 0x15535f5ff700 (LWP 874790)]
[New Thread 0x15535edff700 (LWP 874791)]
[New Thread 0x15535e5ff700 (LWP 874792)]
[New Thread 0x15535ddff700 (LWP 874793)]
[New Thread 0x15535d5ff700 (LWP 874794)]
[New Thread 0x15535cdff700 (LWP 874795)]
[New Thread 0x15535c5ff700 (LWP 874796)]
[New Thread 0x15533bdff700 (LWP 874797)]
[New Thread 0x15533b5ff700 (LWP 874798)]
[New Thread 0x15533adff700 (LWP 874799)]
[New Thread 0x15533a5ff700 (LWP 874800)]
[New Thread 0x155339dff700 (LWP 874801)]
[New Thread 0x1553395ff700 (LWP 874802)]
[New Thread 0x155338dff700 (LWP 874803)]
[New Thread 0x1553385ff700 (LWP 874804)]
[New Thread 0x155317dff700 (LWP 874805)]
[New Thread 0x1553175ff700 (LWP 874806)]
[New Thread 0x155316dff700 (LWP 874807)]
[New Thread 0x1553167ff700 (LWP 874808)]
[New Thread 0x155315dff700 (LWP 874809)]
[New Thread 0x1553155ff700 (LWP 874810)]
[New Thread 0x155314dff700 (LWP 874811)]
[New Thread 0x1553145ff700 (LWP 874812)]
[New Thread 0x1552ebdff700 (LWP 874813)]
[New Thread 0x1552eb7ff700 (LWP 874814)]
[New Thread 0x1552eafff700 (LWP 874815)]
[New Thread 0x1552ea9ff700 (LWP 874816)]
[New Thread 0x1552ea1ff700 (LWP 874817)]
[New Thread 0x1552e99ff700 (LWP 874818)]
[New Thread 0x1552e8fff700 (LWP 874819)]
[New Thread 0x1552e87ff700 (LWP 874820)]
[New Thread 0x1552e3fff700 (LWP 874821)]
[New Thread 0x1552e35ff700 (LWP 874822)]
[New Thread 0x1552e2fff700 (LWP 874823)]
[New Thread 0x1552e27ff700 (LWP 874824)]
[New Thread 0x1552e1fff700 (LWP 874825)]
[New Thread 0x1552e15ff700 (LWP 874826)]
[New Thread 0x1552e0fff700 (LWP 874827)]
[New Thread 0x1552e07ff700 (LWP 874828)]
[New Thread 0x1552a3dff700 (LWP 874829)]
[New Thread 0x1552a37ff700 (LWP 874830)]
[New Thread 0x1552a2fff700 (LWP 874831)]
[New Thread 0x1552a27ff700 (LWP 874832)]
[New Thread 0x1552a1dff700 (LWP 874833)]
[New Thread 0x1552a17ff700 (LWP 874834)]
[New Thread 0x1552a0fff700 (LWP 874835)]
[New Thread 0x1552a07ff700 (LWP 874836)]
[New Thread 0x155283fff700 (LWP 874837)]
[New Thread 0x1552837ff700 (LWP 874838)]

double free or corruption (!prev)

Thread 1 "python" received signal SIGABRT, Aborted.
0x00001555543c952f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install bzip2-libs-1.0.6-28.el8_10.x86_64 glibc-2.28-251.el8_10.14.x86_64 libffi-3.1-24.el8.x86_64 libgcc-8.5.0-24.el8_10.x86_64 libstdc++-8.5.0-24.el8_10.x86_64 libuuid-2.32.1-46.el8.x86_64 libxcrypt-4.1.1-6.el8.x86_64 openssl-libs-1.1.1k-14.el8_6.x86_64 sqlite-libs-3.26.0-19.el8_9.x86_64 xz-libs-5.2.4-4.el8_6.x86_64 zlib-1.2.11-25.el8.x86_64

if I can find a way to run a test job locally that reproduces the issue, I will be able to better control the debugging ...

I can share the code of the function where the issue lies, but I don't know how much it helps without the input

def contourpy2geojson(lon, lat, data, levels, n_decimals=3):
    """using contourpy and shapely, returns a geojson file based on contour plot

    Args:
        lon ([float]): array of longitudes
        lat ([float]): array of latitudes
        data ([float]): data
        levels (dict): array of level: dictionary defined by name (str), min (float), max (float) and color (string)

    Levels example:
        levels = [{
            "name": "0-1",
            "min": 0,
            "max": 1,
            "color": "#fff5f0"
        },{
            "name": "1-2",
            "min": 1,
            "max": 2,
            "color": "#fb6a4a"
        },{
            "name": "2-3",
            "min": 2,
            "max": 3,
            "color": "#67000d"
        }]

    Returns:
        geojson type
    """
    from contourpy import contour_generator
    from shapely import GeometryType, from_ragged_array, set_precision, to_geojson, unary_union
    from shapely import errors as se

    cont_gen = contour_generator(lon, lat, data, fill_type="ChunkCombinedOffsetOffset")

    features = []
    for level in levels:
        filled = cont_gen.filled(level["min"], level["max"])
        points, offsets, outer_offsets = filled[0][0], filled[1][0], filled[2][0]
        if points is not None:
            all_polygons = from_ragged_array(
                GeometryType.POLYGON, points, (offsets, outer_offsets)
            )
            try:
                polygons = set_precision(
                    all_polygons,
                    grid_size=eval(f"1E{-n_decimals}"),
                )
            except se.GEOSException as error:
                print(error)
                valid_polygons = [polygon for polygon in all_polygons if polygon.is_valid]
                polygons = set_precision(
                    valid_polygons,
                    grid_size=eval(f"1E{-n_decimals}"),
                )
            multipolygon = unary_union(polygons)
            feature = {
                "geometry": eval(to_geojson(multipolygon)),
                "type": "Feature",
                "properties": {"fill": level["color"], "level": int(level["name"])},
            }
            # append to list of features
            features.append(feature)

    return {"type": "FeatureCollection", "features": features}

as I said, it does not always crash at the same point in this function, nor with the same exact error

charlienegri · 2025-04-28T13:35:59Z

charlienegri
Apr 28, 2025
Author

data.txt
lat.txt
lon.txt

here a minimal test script to reproduce:

import numpy as np

def contourpy2geojson(lon, lat, data, levels, n_decimals=3):
    """using contourpy and shapely, returns a geojson file based on contour plot

    Args:
        lon ([float]): array of longitudes
        lat ([float]): array of latitudes
        data ([float]): data
        levels (dict): array of level: dictionary defined by name (str), min (float), max (float) and color (string)

    Levels example:
        levels = [{
            "name": "0-1",
            "min": 0,
            "max": 1,
            "color": "#fff5f0"
        },{
            "name": "1-2",
            "min": 1,
            "max": 2,
            "color": "#fb6a4a"
        },{
            "name": "2-3",
            "min": 2,
            "max": 3,
            "color": "#67000d"
        }]


    Returns:
        geojson type

    """
    from contourpy import contour_generator
    from shapely import GeometryType, from_ragged_array, set_precision, to_geojson, unary_union
    from shapely import errors as se

    cont_gen = contour_generator(lon, lat, data, fill_type="ChunkCombinedOffsetOffset")

    features = []
    for level in levels:
        filled = cont_gen.filled(level["min"], level["max"])
        points, offsets, outer_offsets = filled[0][0], filled[1][0], filled[2][0]
        if points is not None:
            all_polygons = from_ragged_array(
                GeometryType.POLYGON, points, (offsets, outer_offsets)
            )
            try:
                polygons = set_precision(
                    all_polygons,
                    grid_size=eval(f"1E{-n_decimals}"),
                )
            except se.GEOSException as error:
                print(error)
                valid_polygons = [polygon for polygon in all_polygons if polygon.is_valid]
                # rejected = [po for po in all_polygons if po not in valid_polygons]
                # print("Rejected polygons:")
                # for rpo in rejected:
                #    print(rpo)
                polygons = set_precision(
                    valid_polygons,
                    grid_size=eval(f"1E{-n_decimals}"),
                )
            multipolygon = unary_union(polygons)
            feature = {
                "geometry": eval(to_geojson(multipolygon)),
                "type": "Feature",
                "properties": {"fill": level["color"], "level": int(level["name"])},
            }
            # append to list of features
            features.append(feature)

    return {"type": "FeatureCollection", "features": features}

levels = [{'name': '1', 'min': 0, 'max': 1, 'color': '#fff5f0'}, {'name': '2', 'min': 1, 'max': 2, 'color': '#fb6a4a'}, {'name': '3', 'min': 2, 'max': 3, 'color': '#67000d'}]

lat = np.loadtxt('lat.txt')
lon = np.loadtxt('lon.txt')
data = np.loadtxt('data.txt')

from itertools import repeat
for i in repeat(None, 10):
    contourpy2geojson(lon, lat, data, levels, n_decimals=3)

calling contourpy2geojson just one time does not cause the issue, calling in repeatedly does

0 replies

jorisvandenbossche · 2025-04-28T21:58:22Z

jorisvandenbossche
Apr 28, 2025
Maintainer

Thanks for providing that reproducer!
Haven't yet looked in more detail, but just to confirm that I also see the segfault with latest shapely with the above script and data, and that I get the following gdb backtrace, pointing to the unary_union method:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352766528) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352766528) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352766528, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c89677 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7ddbb77 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#6  0x00007ffff7ca0cfc in malloc_printerr (str=str@entry=0x7ffff7dde790 "double free or corruption (out)") at ./malloc/malloc.c:5664
#7  0x00007ffff7ca2e70 in _int_free (av=0x7ffff7e1ac80 <main_arena>, p=0x555556244490, have_lock=<optimized out>) at ./malloc/malloc.c:4588
#8  0x00007ffff7ca5453 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9  0x00007fff4944b4d1 in geos::operation::overlayng::OverlayNG::computeEdgeOverlay() () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#10 0x00007fff4944b83e in geos::operation::overlayng::OverlayNG::getResult() () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#11 0x00007fff4944b8ec in geos::operation::overlayng::OverlayNG::overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int, geos::geom::PrecisionModel const*) () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#12 0x00007fff4944c445 in geos::operation::overlayng::OverlayNGRobust::Overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#13 0x00007fff493603b4 in geos::geom::HeuristicOverlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#14 0x00007fff49479673 in geos::operation::geounion::CascadedPolygonUnion::unionActual(geos::geom::Geometry const*, geos::geom::Geometry const*) const () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#15 0x00007fff49479726 in geos::operation::geounion::CascadedPolygonUnion::unionSafe(geos::geom::Geometry const*, geos::geom::Geometry const*) const () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#16 0x00007fff49479873 in geos::operation::geounion::CascadedPolygonUnion::binaryUnion(std::vector<geos::geom::Geometry const*, std::allocator<geos::geom::Geometry const*> > const&, unsigned long, unsigned long) ()
   from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#17 0x00007fff494797f1 in geos::operation::geounion::CascadedPolygonUnion::binaryUnion(std::vector<geos::geom::Geometry const*, std::allocator<geos::geom::Geometry const*> > const&, unsigned long, unsigned long) ()
   from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#18 0x00007fff49479ee6 in geos::operation::geounion::CascadedPolygonUnion::Union() () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#19 0x00007fff4947a163 in geos::operation::geounion::CascadedPolygonUnion::Union(std::vector<geos::geom::Polygon*, std::allocator<geos::geom::Polygon*> >*, geos::operation::geounion::UnionStrategy*) ()
   from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#20 0x00007fff4947f3b3 in geos::operation::geounion::UnaryUnionOp::Union() () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#21 0x00007fff493574a7 in geos::geom::Geometry::Union() const () from /home/joris/conda/envs/shapely-dev/lib/./libgeos.so.3.13.0
#22 0x00007fff4a556260 in GEOSUnaryUnion_r () from /home/joris/conda/envs/shapely-dev/lib/libgeos_c.so.1
#23 0x00007fff4a594d1b in Y_Y_func () from /home/joris/scipy/repos/shapely/shapely/lib.cpython-313-x86_64-linux-gnu.so
#24 0x00007ffff6c07d05 in generic_wrapped_legacy_loop () from /home/joris/conda/envs/shapely-dev/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so
#25 0x00007ffff6c0a805 in try_trivial_single_output_loop () from /home/joris/conda/envs/shapely-dev/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so
#26 0x00007ffff6c120ca in ufunc_generic_fastcall () from /home/joris/conda/envs/shapely-dev/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so
#27 0x00005555555f2643 in PyObject_Call (kwargs={}, args=(<GeometryCollection at remote 0x7fff49636740>,), callable=<numpy.ufunc at remote 0x7fff4a73d140>) at /usr/local/src/conda/python-3.13.2/Objects/call.c:373
#28 PyCFunction_Call (kwargs={}, args=(<GeometryCollection at remote 0x7fff49636740>,), callable=<numpy.ufunc at remote 0x7fff4a73d140>) at /usr/local/src/conda/python-3.13.2/Objects/call.c:381
#29 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.2/Python/generated_cases.c.h:1355

5 replies

charlienegri Apr 29, 2025
Author

thanks for looking into this!
I also managed to get out some more info locally with gdb and interestingly for me bt points to different functions in different runs.... here an example:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352495104) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352495104) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352495104, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c89677 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7ddbb77 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#6  0x00007ffff7ca0cfc in malloc_printerr (str=str@entry=0x7ffff7dd96f1 "corrupted size vs. prev_size") at ./malloc/malloc.c:5664
#7  0x00007ffff7ca17e2 in unlink_chunk (p=p@entry=0x11d4030, av=0x7ffff7e1ac80 <main_arena>) at ./malloc/malloc.c:1629
#8  0x00007ffff7ca2d2b in _int_free (av=0x7ffff7e1ac80 <main_arena>, p=0x11d3820, have_lock=<optimized out>) at ./malloc/malloc.c:4616
#9  0x00007ffff7ca5453 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#10 0x00007fffd49c341c in void std::vector<geos::index::chain::MonotoneChain, std::allocator<geos::index::chain::MonotoneChain> >::_M_realloc_insert<geos::geom::CoordinateSequence const&, unsigned long&, unsigned long&, void*&>(__gnu_cxx::__normal_iterator<geos::index::chain::MonotoneChain*, std::vector<geos::index::chain::MonotoneChain, std::allocator<geos::index::chain::MonotoneChain> > >, geos::geom::CoordinateSequence const&, unsigned long&, unsigned long&, void*&) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#11 0x00007fffd49c32bd in geos::index::chain::MonotoneChainBuilder::getChains(geos::geom::CoordinateSequence const*, void*, std::vector<geos::index::chain::MonotoneChain, std::allocator<geos::index::chain::MonotoneChain> >&) () from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#12 0x00007fffd4a19c3f in geos::noding::MCIndexNoder::computeNodes(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#13 0x00007fffd4a28c79 in geos::noding::snapround::SnapRoundingNoder::addIntersectionPixels(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >&) () from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#14 0x00007fffd4a2c163 in geos::noding::snapround::SnapRoundingNoder::snapRound(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >&, std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >&) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#15 0x00007fffd4a62dea in geos::operation::overlayng::EdgeNodingBuilder::node(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#16 0x00007fffd4a63776 in geos::operation::overlayng::EdgeNodingBuilder::build(geos::geom::Geometry const*, geos::geom::Geometry const*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#17 0x00007fffd4a75edf in geos::operation::overlayng::OverlayNG::computeEdgeOverlay() ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#18 0x00007fffd4a765bd in geos::operation::overlayng::OverlayNG::getResult() ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#19 0x00007fffd4a7c414 in geos::operation::overlayng::PrecisionReducer::reducePrecision(geos::geom::Geometry const*, geos::geom::PrecisionModel const*, bool) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#20 0x00007fffd4ad40e6 in geos::precision::PrecisionReducerTransformer::reduceArea(geos::geom::Geometry const*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#21 0x00007fffd4ad40fa in geos::precision::PrecisionReducerTransformer::transformMultiPolygon(geos::geom::MultiPolygon const*, geos::geom::Geometry const*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#22 0x00007fffd49ab53f in geos::geom::util::GeometryTransformer::transform(geos::geom::Geometry const*) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#23 0x00007fffd4ad3608 in geos::precision::PrecisionReducerTransformer::reduce(geos::geom::Geometry const&, geos::geom::PrecisionModel const&, bool) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#24 0x00007fffd4ad0b71 in geos::precision::GeometryPrecisionReducer::reduce(geos::geom::Geometry const&) ()
   from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#25 0x00007ffff2cb85af in GEOSGeom_setPrecision_r () from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos_c-abcdd5fa.so.1.19.2
#26 0x00007ffff2d08601 in set_precision_func (args=0x7fffffffaea0, dimensions=0x7fffffffad88, steps=0x7fffffffada0, data=<optimized out>) at src/ufuncs.c:2801
#27 0x00007ffff6a4ced5 in generic_wrapped_legacy_loop () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#28 0x00007ffff6a4ffbe in try_trivial_single_output_loop () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#29 0x00007ffff6a5a573 in ufunc_generic_fastcall () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#30 0x0000000000435a3f in PyVectorcall_Call (callable=callable@entry=<numpy.ufunc at remote 0x7ffff36f9bf0>, 
    tuple=tuple@entry=(<numpy.ndarray at remote 0x7ffff039c740>, <float at remote 0x7ffff03e90f0>, <numpy.int32 at remote 0x7ffff436ebf0>), kwargs=kwargs@entry={})
    at ../Objects/call.c:255
#31 0x0000000000435d08 in _PyObject_Call (tstate=0xa3f150, callable=<numpy.ufunc at remote 0x7ffff36f9bf0>, 
    args=(<numpy.ndarray at remote 0x7ffff039c740>, <float at remote 0x7ffff03e90f0>, <numpy.int32 at remote 0x7ffff436ebf0>), kwargs={}) at ../Objects/call.c:290
#32 0x0000000000435d71 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:317
#33 0x00000000004ed390 in do_call_core (tstate=0xa3f150, trace_info=0x7fffffffcd70, func=<numpy.ufunc at remote 0x7ffff36f9bf0>, 
    callargs=(<numpy.ndarray at remote 0x7ffff039c740>, <float at remote 0x7ffff03e90f0>, <numpy.int32 at remote 0x7ffff436ebf0>), kwdict={}) at ../Python/ceval.c:5945
#34 0x00000000004e82a9 in _PyEval_EvalFrameDefault (tstate=0xa3f150, 
    f=Frame 0x7ffff36718b0, for file /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/_geometry.py, line 959, in set_precision (geometry=<numpy.ndarray at remote 0x7ffff039c740>, grid_size=<float at remote 0x7ffff03e90f0>, mode=0, kwargs={}), throwflag=<optimized out>) at ../Python/ceval.c:4277
#35 0x00000000004eb409 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7ffff36718b0, for file /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/_geometry.py, line 959, in set_precision (geometry=<numpy.ndarray at remote 0x7ffff039c740>, grid_size=<float at remote 0x7ffff03e90f0>, mode=0, kwargs={}), tstate=0xa3f150) at ../Include/internal/pycore_ceval.h:46
#36 _PyEval_Vector (tstate=0xa3f150, con=0x7ffff36d6f20, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5067
#37 0x000000000042f7d6 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#38 0x0000000000435b03 in PyVectorcall_Call (callable=callable@entry=<function at remote 0x7ffff36d6f10>, tuple=tuple@entry=(<numpy.ndarray at remote 0x7ffff039c740>,), 
    kwargs=kwargs@entry={'grid_size': <float at remote 0x7ffff03e90f0>}) at ../Objects/call.c:267
#39 0x0000000000435d08 in _PyObject_Call (tstate=0xa3f150, callable=<function at remote 0x7ffff36d6f10>, args=(<numpy.ndarray at remote 0x7ffff039c740>,), 
    kwargs={'grid_size': <float at remote 0x7ffff03e90f0>}) at ../Objects/call.c:290
#40 0x0000000000435d71 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:317

charlienegri Apr 29, 2025
Author

and another with double free or corruption (out) like yours

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352495104) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352495104) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352495104, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c89677 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7ddbb77 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#6  0x00007ffff7ca0cfc in malloc_printerr (str=str@entry=0x7ffff7dde790 "double free or corruption (out)") at ./malloc/malloc.c:5664
#7  0x00007ffff7ca2e70 in _int_free (av=0x7ffff7e1ac80 <main_arena>, p=0x11aaa70, have_lock=<optimized out>) at ./malloc/malloc.c:4588
#8  0x00007ffff7ca5453 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9  0x00007fffd4992ed4 in geos::geom::LineString::~LineString() () from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos-3ef06f11.so.3.13.1
#10 0x00007ffff2cc3b31 in geos::geom::Polygon::~Polygon() () from /home/charlien/v/cams235b/lib/python3.10/site-packages/shapely/../shapely.libs/libgeos_c-abcdd5fa.so.1.19.2
#11 0x00007ffff2cfe9a1 in GeometryObject_dealloc (self=0x7fffe8f3d040) at src/pygeom.c:63
#12 0x0000000000481bef in subtype_dealloc (self=<Polygon at remote 0x7fffe8f3d040>) at ../Objects/typeobject.c:1460
#13 0x0000000000467705 in _Py_Dealloc (op=<optimized out>) at ../Objects/object.c:2301
#14 0x00007ffff6982c3b in clear_object_strided_loop () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#15 0x00007ffff69c626a in PyArray_ClearArray () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#16 0x00007ffff693c088 in array_dealloc () from /home/charlien/v/cams235b/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#17 0x0000000000467705 in _Py_Dealloc (op=<optimized out>) at ../Objects/object.c:2301
#18 0x00000000004596e8 in _Py_DECREF (op=<optimized out>, lineno=567, filename=0x70e1d4 "../Include/object.h") at ../Include/object.h:500
#19 _Py_XDECREF (op=<optimized out>) at ../Include/object.h:567
#20 insertdict (mp=mp@entry=0x7ffff43a3e30, key=key@entry='all_polygons', hash=-7085365537282132168, value=value@entry=<numpy.ndarray at remote 0x7fffee39c640>)
    at ../Objects/dictobject.c:1145

jorisvandenbossche May 2, 2025
Maintainer

Indeed, it's not always the same backtrace, running it now again it also points to something in OverlayNG, but in the end also is doing some free, so it might indeed be related to memory management / how the geometry was created.

From there, I could reduce the code example down to just creating the polygons from ragged arrays:

import numpy as np
import shapely

points = np.random.randn(1085, 2)
offsets = np.array([   0,  237,  582,  639,  868,  937, 1085], dtype=np.uint32)
outer_offsets = np.array([0, 1, 5, 6], dtype=np.uint32)

polygons = shapely.from_ragged_array(
    shapely.GeometryType.POLYGON, points, (offsets, outer_offsets)
)

this segfaults when exiting the python session, i.e. when deallocating the objects/data.

jorisvandenbossche May 2, 2025
Maintainer

I have a fix for the above reproduced, and also confirmed your original script no longer segfaults with that: #2291

@charlienegri and thanks again for reporting!

charlienegri May 2, 2025
Author

thanks and excellent! 👍

memory issues with 2.1.0 #2284

Uh oh!

charlienegri Apr 25, 2025

Replies: 3 comments · 6 replies

Uh oh!

jorisvandenbossche Apr 25, 2025 Maintainer

Uh oh!

Uh oh!

charlienegri Apr 28, 2025 Author

Uh oh!

charlienegri Apr 28, 2025 Author

Uh oh!

jorisvandenbossche Apr 28, 2025 Maintainer

Uh oh!

Uh oh!

charlienegri Apr 29, 2025 Author

Uh oh!

charlienegri Apr 29, 2025 Author

Uh oh!

jorisvandenbossche May 2, 2025 Maintainer

Uh oh!

jorisvandenbossche May 2, 2025 Maintainer

Uh oh!

charlienegri May 2, 2025 Author

charlienegri
Apr 25, 2025

Replies: 3 comments 6 replies

jorisvandenbossche
Apr 25, 2025
Maintainer

charlienegri Apr 28, 2025
Author

charlienegri
Apr 28, 2025
Author

jorisvandenbossche
Apr 28, 2025
Maintainer

charlienegri Apr 29, 2025
Author

charlienegri Apr 29, 2025
Author

jorisvandenbossche May 2, 2025
Maintainer

jorisvandenbossche May 2, 2025
Maintainer

charlienegri May 2, 2025
Author