Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: Use 2GiB chunking code for fwrite() on mingw32/64 #23505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 17, 2023
Merged

BUG: Use 2GiB chunking code for fwrite() on mingw32/64 #23505

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4042674
BUG: Use 2GiB chunking code for fwrite() on mingw32/64
cbrt64 Mar 30, 2023
1c17794
MAINT: Use fwrite chunking generally on Windows 64 and size_t as type
seberg May 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion numpy/core/include/numpy/npy_common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@
#define NPY_STEALS_REF_TO_ARG(n)
#endif

/* 64 bit file position support, also on win-amd64. Ticket #1660 */
/* 64 bit file position support, also on win-amd64. Issue gh-2256 */
#if defined(_MSC_VER) && defined(_WIN64) && (_MSC_VER > 1400) || \
defined(__MINGW32__) || defined(__MINGW64__)
#include <io.h>
Expand Down
19 changes: 13 additions & 6 deletions numpy/core/src/multiarray/convert.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,18 @@ PyArray_ToFile(PyArrayObject *self, FILE *fp, char *sep, char *format)
size = PyArray_SIZE(self);
NPY_BEGIN_ALLOW_THREADS;

#if defined (_MSC_VER) && defined(_WIN64)
/* Workaround Win64 fwrite() bug. Issue gh-2556
#if defined(NPY_OS_WIN64)
/*
* Workaround Win64 fwrite() bug. Issue gh-2256
* The native 64 windows runtime has this issue, the above will
* also trigger UCRT (which doesn't), so it could be more precise.
*
* If you touch this code, please run this test which is so slow
* it was removed from the test suite
* it was removed from the test suite. Note that the original
* failure mode involves an infinite loop during tofile()
*
* import tempfile, numpy as np
* from numpy.testing import (assert_)
* fourgbplus = 2**32 + 2**16
* testbytes = np.arange(8, dtype=np.int8)
* n = len(testbytes)
Expand All @@ -177,16 +184,16 @@ PyArray_ToFile(PyArrayObject *self, FILE *fp, char *sep, char *format)
* assert_((a[-n:] == testbytes).all())
*/
{
npy_intp maxsize = 2147483648 / PyArray_DESCR(self)->elsize;
npy_intp chunksize;
size_t maxsize = 2147483648 / (size_t)PyArray_DESCR(self)->elsize;
size_t chunksize;

n = 0;
while (size > 0) {
chunksize = (size > maxsize) ? maxsize : size;
n2 = fwrite((const void *)
((char *)PyArray_DATA(self) + (n * PyArray_DESCR(self)->elsize)),
(size_t) PyArray_DESCR(self)->elsize,
(size_t) chunksize, fp);
chunksize, fp);
if (n2 < chunksize) {
break;
}
Expand Down