Use BigInt for shape, maxshape and total_size in metadata #112
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
I made a big file to test slicing like this: import numpy as np
import h5py
N = 1000
data = np.empty((N, N, N), dtype="float32")
x = np.arange(N, dtype="float32")
y = x * N
z = x / N
data += z[None, None, :]
data += y[None, :, None]
data += x[:, None, None]
with h5py.File("big.h5", "w") as output:
output.create_dataset("data", data=data)The resulting file is 3.8 GiB, and I am able to slice it without problems. I think slicing would have worked fine for very large files even without this change, as the Javascript MAX_SAFE_INTEGER is huge already We could probably use the plain Javascript |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes:
shape: number[]=>shape: bigint[]maxshape: number[]=>maxshape: bigint[]total_size: number=>total_size: bigintcheck_malloc(nbytes: number | bigint): number;that returns a pointer after checkingBoth of these changes should help when interacting with large datasets (> 2GB), and should address #111
You still won't be able to read such a dataset directly into memory without slicing (you'll run into the memory limit, but hopefully with a helpful error message), but hopefully you'll be able to slice such large datasets successfully now. It seems like it would have been difficult/impossible to address dataset regions with offsets > 2GB in the slice function, before these changes, e.g. calculating the offset and strides might have failed.