gh-120754: Ensure _stat_atopen is cleared on fd change #125166

cmaloney · 2024-10-08T23:51:06Z

While adding _stat_atopen I accidentally created a leak in the C implementation because the _stat_atopen member was set to NULL in the C implementation, but the memory wasn't freed. That was resolved, but it showed there were a number of potential cases where _stat_atopen should be cleared (the information is out of date because the fd changed) that it was not. In this PR I audited by reading through FileIO (in _io and _pyio), and in all cases where the fd is changed/closed made sure to clear/reset _stat_atopen as well.

Followup from: #124225 (comment), cc: @vstinner

Issue: Speed up open().read() pattern by reducing the number of system calls #120754

Performed an audit of `fileio.c` and `_pyio` and made sure anytime the fd changes the stat result, if set, is also cleared/changed. There's one case where it's not cleared, if code would clear it in __init__, keep the memory allocated and just do another fstat with the existing memory.

vstinner · 2024-10-09T07:53:50Z

Modules/_io/fileio.c

-            if (internal_close(self) < 0)
+            /* Have to close the existing file first.
+
+               This leaves the stat so we can reuse the memory. */


I don't think that this optimization is worth it, I would prefer to always clear the cache in close().

cmaloney · 2024-10-09T18:46:33Z

Modules/_io/fileio.c

@@ -455,11 +458,17 @@ _io_FileIO___init___impl(fileio *self, PyObject *nameobj, const char *mode,
 #endif
    }

-    PyMem_Free(self->stat_atopen);


Not sure which is preferred here. The Free + New back to back feels weird, but version with the conditional feels like "extra work/code that should never be executed" (but that the compiler / tools that run currently won't validate...).

Maybe if (ptr != NULL) free(ptr) would be more regular and avoid the 4 lines long comment.

Moved to that, dropped comment and simplified / back to always calling PyMem_New

vstinner

There are compiler warnings that you can see in the review tab.

cmaloney · 2024-10-09T20:41:43Z

I think the latest commit fixes the compiler warnings (was a testing/debugging assert I accidentally committed with a missing ;, latest changes remove that assert). Not able to find them in the latest changes.

Modules/_io/fileio.c

cmaloney · 2024-10-30T05:51:50Z

CI "Tests / Ubuntu SSL tests with OpenSSL" failed because of what looks like a network hang / not these changes: Received 164773440 of 168967744 (97.5%), 0.0 MBs/sec while doing "Configure ccache action"

vstinner · 2024-10-30T08:52:30Z

Modules/_io/fileio.c

@@ -178,6 +180,8 @@ _io_FileIO_close_impl(fileio *self, PyTypeObject *cls)
            PyErr_Clear();
        }
    }
+    PyMem_Free(self->stat_atopen);
+    self->stat_atopen = NULL;


internal_close() already does the same, so it's redundant, no?

Tradeoff for matching what _pyio does which does os.close() in these cases vs an internal_close that does the OS close + other cleanup. Happy to drop though / definitely feels redundant to me in these two cases.

vstinner · 2024-10-30T08:53:09Z

Modules/_io/fileio.c

@@ -266,10 +270,13 @@ _io_FileIO___init___impl(fileio *self, PyObject *nameobj, const char *mode,
    assert(PyFileIO_Check(state, self));
 #endif
    if (self->fd >= 0) {
+        PyMem_Free(self->stat_atopen);
+        self->stat_atopen = NULL;


internal_close() already does the same, so it's redundant, no?

Moderately, in the self->closefd is false case behavior would differ from _pyio, but that will be caught by the PyMem_Free + PyMem_New below, so think this is always redundant / not needed

vstinner · 2024-10-30T08:54:29Z

Modules/_io/fileio.c

@@ -455,7 +462,9 @@ _io_FileIO___init___impl(fileio *self, PyObject *nameobj, const char *mode,
 #endif
    }

-    PyMem_Free(self->stat_atopen);
+    if (self->stat_atopen != NULL) {


PyMem_Free(NULL) is well defined: it does nothing, so you might omit the if (stat_atopen != NULL) test.

vstinner · 2024-11-01T21:51:11Z

Merged, thank you.

…n#125166) Performed an audit of `fileio.c` and `_pyio` and made sure anytime the fd changes the stat result, if set, is also cleared/changed. There's one case where it's not cleared, if code would clear it in __init__, keep the memory allocated and just do another fstat with the existing memory.

cmaloney added 2 commits October 8, 2024 16:41

pythongh-120754: _pyio ensure _stat_atopen is cleared on fd change

d2cb365

bedevere-app bot added the awaiting review label Oct 8, 2024

bedevere-app bot mentioned this pull request Oct 8, 2024

Speed up open().read() pattern by reducing the number of system calls #120754

Closed

_io: Add curlies since already modifying if

854ae91

vstinner reviewed Oct 9, 2024

View reviewed changes

cmaloney added 2 commits October 9, 2024 11:42

_io: Always clear self->stat_atopen

c6da86e

pyio: Always clear stat_atopen just before close

0f64483

cmaloney commented Oct 9, 2024

View reviewed changes

cmaloney added 2 commits October 9, 2024 11:47

Remove assert accidentally commited

e2eebb0

_io: fix conditional

5299a04

vstinner reviewed Oct 9, 2024

View reviewed changes

vstinner reviewed Oct 10, 2024

View reviewed changes

Modules/_io/fileio.c Show resolved Hide resolved

cmaloney and others added 3 commits October 22, 2024 14:40

Clear before internal_close to match _pyio os.close callsites

d749709

Move to conditional free, simplify code, drop comment

b429587

Merge branch 'main' into cmaloney/fd_cleanup

f08defa

vstinner reviewed Oct 30, 2024

View reviewed changes

Apply review fixes

e153d97

vstinner added the skip news label Nov 1, 2024

vstinner merged commit 72dd471 into python:main Nov 1, 2024
39 checks passed

bedevere-app bot removed the awaiting review label Nov 1, 2024

cmaloney deleted the cmaloney/fd_cleanup branch November 1, 2024 21:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-120754: Ensure _stat_atopen is cleared on fd change #125166

gh-120754: Ensure _stat_atopen is cleared on fd change #125166

cmaloney commented Oct 8, 2024 •

edited by bedevere-app bot

Loading

vstinner Oct 9, 2024

cmaloney Oct 9, 2024

vstinner Oct 29, 2024

cmaloney Oct 30, 2024

vstinner left a comment

cmaloney commented Oct 9, 2024 •

edited

Loading

cmaloney commented Oct 30, 2024

vstinner Oct 30, 2024

cmaloney Oct 31, 2024 •

edited

Loading

vstinner Oct 30, 2024

cmaloney Oct 31, 2024

vstinner Oct 30, 2024

vstinner commented Nov 1, 2024

gh-120754: Ensure _stat_atopen is cleared on fd change #125166

gh-120754: Ensure _stat_atopen is cleared on fd change #125166

Conversation

cmaloney commented Oct 8, 2024 • edited by bedevere-app bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner left a comment

Choose a reason for hiding this comment

cmaloney commented Oct 9, 2024 • edited Loading

cmaloney commented Oct 30, 2024

Choose a reason for hiding this comment

cmaloney Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented Nov 1, 2024

cmaloney commented Oct 8, 2024 •

edited by bedevere-app bot

Loading

cmaloney commented Oct 9, 2024 •

edited

Loading

cmaloney Oct 31, 2024 •

edited

Loading