Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: load no longer reads files with shape=(-1,...) #29217

Open
@defjaf

Description

@defjaf

Describe the issue:

Prior to numpy 2.3.0, load (and its relatives) could read a *.npy file with shape[0]=-1, and correctly discover the number of rows in the file from its length. (The documentation is silent on the issue.) In 2.3.0, due to commit d8cba36, there is a check that the product of the entries in shape is equal to the number of entries in the file, which fails for this case.

I think this bug/regression can be fixed by only checking that the count matches the actual length when shape[0]!=-1 (and possibly adding a check that all entries in the shape must be positive except for that). (I have never contributed, but am happy to try to raise a PR for this.)

Reproduce the code example:

import numpy as np
arr = np.load("badshape.npy")

## I cannot attach a *.npy file as an example

Error message:

File [~/.venv/lib/python3.13/site-packages/numpy/lib/_format_impl.py:874](http://localhost:8888/~/.venv/lib/python3.13/site-packages/numpy/lib/_format_impl.py#line=873), in read_array(fp, allow_pickle, pickle_kwargs, max_header_size)
    870             array[i:i + read_count] = numpy.frombuffer(data, dtype=dtype,
    871                                                      count=read_count)
    873 if array.size != count:
--> 874     raise ValueError(
    875         "Failed to read all data for array. "
    876         f"Expected {shape} = {count} elements, "
    877         f"could only read {array.size} elements. "
    878         "(file seems not fully written?)"
    879     )
    881 if fortran_order:
    882     array.shape = shape[::-1]

ValueError: Failed to read all data for array. Expected (-1, 1) = -1 elements, could only read 1000 elements. (file seems not fully written?)

Python and NumPy Versions:

2.3.0
3.13.5 (main, Jun 11 2025, 15:36:57) [Clang 17.0.0 (clang-1700.0.13.3)]

Runtime Environment:

No response

Context for the issue:

I think that numpy itself never uses this "feature" but we have some home-grown npy-file writing code which does (and is useful so that we can append rows to the file without updating the header).

Metadata

Metadata

Assignees

No one assigned

    Labels

    01 - Enhancement57 - Close?Issues which may be closable unless discussion continued

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions