This is just arr.flags.fnc

That's cool, update on the way.

This doesn't do anything; del just deletes the local reference to self, and won't actually call __del__ here.

That's bad, I'll also fix this in the npy-append-array module, thanks a lot!

This can never fire, since the check above already checks for this,

Indeed, I've missed that, will also update the npy-append-array module.

Instead of doing this dance, could you just call _read_array_header(fp, version), and then use fp.tell() to work out the header length?

Indeed, since now I have access to all the np.lib.format functions, I can do this directly, very nice!

Why not just use self.header_length?

Indeed, that length should never change.

I've just double checked: header_length should probably not be an attribute of NpyAppendArray but rather a local variable of __init and I should also rename it, update on the way.

What are 10 and 12 here?

This is the byte offset. The header length is a little-endian unsigned short int, which has two bytes in version 1 and 4 bytes in version 2. This is where the 10 and 12 are from. I can replace this with a calculation like in
https://numpy.org/devdocs/reference/generated/numpy.lib.format.html#format-version-1-0

Should disappear in the new version.

I think you should add an argument to write_array_header_2_0 to encapsulate this, either as:

• min_header_size, which sets the minimum header padding size
• extra_header_padding, which augments the header with extra padding.

I think the extra_header_padding is what we need. The question is whether to make it an argument or just add 64 byte by default, since as mentioned, more energy would be needed to populate such an array than would be necessary to boil earth's oceans. In a distant future, when our descendants eventually build a dyson swarm, this value may be increased to 128 byte to account for all particles in the known universe.
Adding 64 byte by default will increase the size of every .npy file by 64 bytes, but I cannot really judge how much of a problem that would be. Maybe none at all, maybe filesystem overhead adds 64 byte anyway, maybe this would pose some issues to certain users that have millions/billions of .npy files laying around. What do you think?

more energy would be needed to populate such an array than would be necessary to boil earth's oceans. In a distant future, when our descendants eventually build a dyson swarm, this value may be increased to 128 byte to account for all particles in the known universe.

I think you're thinking about this the wrong way. The length of an array is stored in an int64, so we're already limited to 2**63-1. The question is what the longest string is that is less than that size; for instance, (9223372036854775807,) is shorter than (10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10). On the other hand, I think the way you have appending set up means that only one dimension of the array can grow, in which case len(str(2**63-1)) - len(str(len(arr))) is the needed amount of padding

So perhaps write_array_header_2_0(..., extra_padding=True) and have write_array_header_2_0 deal with calculating how much padding is needed.

I have chosen to try the -1 option, see below. Then we would not need to handle possible growing headers in the first place.

That's true, the more dimensions and larger dimensions the array has, the more bytes are covered by advancing along the first axis. So the worst case is an array with only one dimension of dtype int8 or so. So len(str(2**63-1)) (=19) would be an upper bound. That means we could also just take 19 bytes as a buffer instead of 64 bytes. I would propose to make the code as simple as possible, so one could just add 19 bytes and not try to reduce that byte count by taking into account the influence of the other axes. For most arrays 19 spare bytes would probably not even make a difference in the file size, although even if, it would probably be consumed by the filesystem overhead.
I'll do it in a follow up pull request, first this pull request here (which does not affect anything else) should go through (as far as I understand it).

Actually, shouldn't it be 2^64-1? I mean size_t is a uint64_t in C (for 64 bit machines). This would increase the buffer from 19 to 20 bytes.

This doesn't quite feel safe to me; you can't guarantee in python that the __del__ or close method is actually called, and you can risk leaving behind a silently truncated file without any record that it got truncated.

Should we have a "dirty" flag in the header, which gets cleared by close() such that the user can at least detect when they open a file that's broken in this way?

This issue used to be around before as well, also in numpy.save: if for some reason the program crashes before the file is finished, it header and content will be in an inconsistent state. This was also an issue in all previous versions of AppendArray but may be more pronounced here.

Did I get you right that you supposed to add a dirty flag to the .npy file format? I was trying to add some functionality without introducing a new numpy file format version. However, I already had some ideas for a follow up pull request, which would allow -1 to be specified as shape[0], so that the size of the array could be inferred from the file size. This would be consistent with e.g. numpy.reshape. Would that eliminate the need for a dirty flag, what do you think?

So, here are the options as I see them:

• Rewrite the header after every append call. This probably slows things down a bit as you end up jumping back and forth in the file. This at least means that as long as append itself is not interrupted, the user knows their data is safe. Perhaps we should profile it.
• Write -1 to the header while the AppendArray is open, then replace it with the final size when .close() runs normally. This is essentially the dirty flag I was suggesting, without needing any new header space. Old versions of numpy will not be able to open a file that is dirty in this way.
• Add "dirty": True to the header dictionary while the AppendArray is open, and then remove it again

Option 2, the -1 as shape[0] of the array I have meant in a different way: the -1 would basically stay there forever, the header would not need to be touched anymore and would also never grow. Having the -1 there indicates that the size of the array is determined by the file size. So when the array is loaded and the actual ndarray constructed, the shape of the ndarray would not contain -1 but the actual size in the ndarray object, while the .npy file header stays the same. This would allow .npy to be the digital version of CSV (just kidding, I mean "binary version of CSV" of course), where data is also simply appended to the file's end. Perfectly suited for binary log files.
There would not be any indication whether a write has succeeded or failed, though. A partial, unsuccessful write could however be potentially identified if the file size (minus header size) is not divisible by the multiplied shape entries (times entry byte size). So potential errors need to be caught elsewhere in the file creation process.
All .npy files could theoretically have their shape[0] of -1, since it should be possible to derive the array size from the file size anyway.
What do you think about this interpretation of option 2?

I cannot speak for fortran_order though. Does anybody have a clue?

I have implemented option 1, "Rewrite the header after every append call" and then option 2 as described in a later commit. What do you think? I can revert to the option 1 commit if option 2 goes too far, as it slightly changes the .npy format.

Also, I made some assumptions about how the fortran part might work. If someone had a fortran array to actually test AppendArray, that would be great.

def func(self): None is not a valid expression, so this will have to be updated:

- class AppendArray:
-     def __init__(self, filename): None
-     def append(self, arr): None
+ import os
+ from numpy.typing import NDArray
+
+ class AppendArray:
+     def __init__(self, filename: str | bytes | os.PathLike[str | bytes]) -> None: ...
+     def append(self, arr: NDArray[Any]) -> None: ...

Secondly, a number of (public) methods and attributes are still absent from the stub file (__enter__, fp, etc.),
can add those as well?

Yes, that would be great, I have simply forgotten it in a hurry. Looks much nicer (and useful) with type annotations as well. Does it make sense to make fp public? Does that fit with the numpy style? Otherwise I'd simply make everything private but special methods, append and close. How does the modification work technically? Will it change the pull request? Can I pull it back into my forked repository?

I think I've figured out, this is just a comment with diff syntax highlighting. Just changed it in my repo.

I'd rather fix test_large_header() instead.

Yes, I thought this might be an option too, this is why I've mentioned it in the comment. I'll have a look into it (next push won't fix it, just minor cleanup).

Just that I understand correctly: fixing test_large_header() implies that _write_array_header() can assume that the array is correct enough which implies it must do the same sanity checks e.g. as _read_array_header(), which would then imply that shape and fortran_order is set (otherwise it would not make sense to modify test_large_header() and _write_array_header() in the first place).
Enabling sanity checks for _write_array_header() would need the tests test_large_header(), test_bad_header() and test_metadata_dtype() to be modified as well. While the first one is straight forward, the second one a little ugly and limits the readability of the test case slightly and could make the test case useless in case of future format modifications, the third one is a little challenging because _write_array_header() cannot produce certain kinds of invalid arrays anymore.
I've pushed with the test_metadata_dtype() failing.
I see three options to resolve the situation:

What do you think?

I think I might just have found a middle ground by skipping the sanity test in _write_header and only assuming that shape and fortran_order are set and modifying tests only so that this works out.

I think this could be it, what do you think?

There's already a from io import BytesIO import at the top level. No need for a local import.

Done

I would recommend constructing test dicts directly instead of grabbing them from a real array. Then you will be more able to test arbitrary lengths of the growth axis, and thus test the important property of the feature introduced here: the invariance of the header length under such changes.

That's is a great idea, just implemented this.