From 3df957c5d7c85abefb905886ef600fdcc4d7ef00 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Thu, 22 Sep 2022 00:30:39 +0200 Subject: [PATCH 01/25] Brought the same changes as in now closed PR 22431, but to the latest version of main --- Doc/library/tempfile.rst | 75 +++++++++++++++---- Doc/whatsnew/3.12.rst | 5 ++ Lib/tempfile.py | 31 ++++++-- Lib/test/test_tempfile.py | 59 +++++++++++++++ .../2020-09-28-04-56-04.bpo-14243.YECnxv.rst | 1 + 5 files changed, 150 insertions(+), 21 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index b7e604c1b70acb..5914731470ff9b 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -75,20 +75,50 @@ The module defines the following user-callable items: Added *errors* parameter. -.. function:: NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, delete=True, *, errors=None) - - This function operates exactly as :func:`TemporaryFile` does, except that - the file is guaranteed to have a visible name in the file system (on - Unix, the directory entry is not unlinked). That name can be retrieved - from the :attr:`name` attribute of the returned - file-like object. Whether the name can be - used to open the file a second time, while the named temporary file is - still open, varies across platforms (it can be so used on Unix; it cannot - on Windows). If *delete* is true (the default), the file is - deleted as soon as it is closed. +.. function:: NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, delete=True, *, errors=None, delete_on_close=True) + + This function operates exactly as :func:`TemporaryFile` does, except the + following differences: + + * The file is guaranteed to have a visible name in the file system (on Unix, + the directory entry is not unlinked). + + * There is more granularity in the deletion behaviour of the file (see + ``delete_on_close`` below) + The returned object is always a file-like object whose :attr:`!file` - attribute is the underlying true file object. This file-like object can - be used in a :keyword:`with` statement, just like a normal file. + attribute is the underlying true file object. This file-like object can be + used in a :keyword:`with` statement, just like a normal file. The name of the + temporary file can be retrieved from the :attr:`name` attribute of the + returned file-like object. + + If *delete* is true (the default) and *delete_on_close* is true (the + default), the file is deleted as soon as it is closed. If *delete* is true + and *delete_on_close* is false, the file is deleted on context manager exit + only. If *delete* is false, the value of *delete_on_close* is ignored. + + While the named temporary file is open, the file can always be opened again + on POSIX. On Windows, it can be opened again if *delete* is false, or if + *delete_on_close* is false, or if the additional open shares delete access + (e.g. by calling :func:`os.open` with the flag ``O_TEMPORARY``). On + Windows, if *delete* is true and *delete_on_close* is false, additional + opens that do not share delete access (e.g. via builtin :func:`open`) must + be closed before exiting the context manager, else the :func:`os.unlink` + call on context manager exit will fail with a :exc:`PermissionError`. + + To use the name of the temporary file to open the closed file second time, + either make sure not to delete the file upon closure (set the *delete* + parameter to be false) or, in case the temporary file is created in a + :keyword:`with` statement, set the *delete_on_close* parameter to be false. + The latter approach is recommended as it provides assistance in automatic + cleaning of the temporary file upon the context manager exit. + + On Windows, if *delete_on_close* is false, and the file is created in a + directory for which the user lacks delete access, then the :func:`os.unlink` + call on exit of the context manager will fail with a :exc:`PermissionError`. + This cannot happen when *delete_on_close* is true because delete access is + requested by the open, which fails immediately if the requested access is not + granted. On POSIX (only), a process that is terminated abruptly with SIGKILL cannot automatically delete any NamedTemporaryFiles it created. @@ -98,6 +128,9 @@ The module defines the following user-callable items: .. versionchanged:: 3.8 Added *errors* parameter. + .. versionchanged:: 3.12 + Added *delete_on_close* parameter. + .. class:: SpooledTemporaryFile(max_size=0, mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, *, errors=None) @@ -346,6 +379,22 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # file is now closed and removed + # create a temporary file using a context manager, note the name, + # close the file, use the name to open the file again + >>> with tempfile.TemporaryFile(delete_on_close=False) as fp: + ... fp.write(b'Hello world!') + ... fp_name = fp.name + ... fp.close() + # the file is closed, but not removed + # open the file again by using its name + ... f = open(fp_name) + ... f.seek(0) + ... f.read() + b'Hello world!' + ... f.close() + >>> + # file is now removed + # create a temporary directory using the context manager >>> with tempfile.TemporaryDirectory() as tmpdirname: ... print('created temporary directory', tmpdirname) diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 3fbc7b63370a3e..8f3d494fda64a8 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -142,6 +142,11 @@ unicodedata * The Unicode database has been updated to version 15.0.0. (Contributed by Benjamin Peterson in :gh:`96734`). +tempfile +-------- + +The :class:`tempfile.NamedTemporaryFile` class has a new optional parameter +*delete_on_close* (Contributed by Evgeny Zorin in :gh:`58451`.) Optimizations ============= diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 480c17232e9f09..b2b05586902a82 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -421,10 +421,11 @@ class _TemporaryFileCloser: file = None # Set here since __del__ checks it close_called = False - def __init__(self, file, name, delete=True): + def __init__(self, file, name, delete=True, delete_on_close=True): self.file = file self.name = name self.delete = delete + self.delete_on_close = delete_on_close # NT provides delete-on-close as a primitive, so we don't need # the wrapper to do anything special. We still use it so that @@ -442,7 +443,7 @@ def close(self, unlink=_os.unlink): try: self.file.close() finally: - if self.delete: + if self.delete and self.delete_on_close: unlink(self.name) # Need to ensure the file is deleted on __del__ @@ -464,11 +465,13 @@ class _TemporaryFileWrapper: remove the file when it is no longer needed. """ - def __init__(self, file, name, delete=True): + def __init__(self, file, name, delete=True, delete_on_close=True): self.file = file self.name = name self.delete = delete - self._closer = _TemporaryFileCloser(file, name, delete) + self.delete_on_close = delete_on_close + self._closer = _TemporaryFileCloser(file, name, delete, + delete_on_close) def __getattr__(self, name): # Attribute lookups are delegated to the underlying file @@ -500,6 +503,15 @@ def __enter__(self): def __exit__(self, exc, value, tb): result = self.file.__exit__(exc, value, tb) self.close() + # If the file is to be deleted, but not on close, delete it now. + if self.delete and not self.delete_on_close: + try: + _os.unlink(self.name) + # It is okay to ignore FileNotFoundError. The file may have + # been deleted already. + except FileNotFoundError: + pass + return result def close(self): @@ -521,7 +533,8 @@ def __iter__(self): def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, - dir=None, delete=True, *, errors=None): + dir=None, delete=True, *, errors=None, + delete_on_close=True): """Create and return a temporary file. Arguments: 'prefix', 'suffix', 'dir' -- as for mkstemp. @@ -529,7 +542,9 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, 'buffering' -- the buffer size argument to io.open (default -1). 'encoding' -- the encoding argument to io.open (default None) 'newline' -- the newline argument to io.open (default None) - 'delete' -- whether the file is deleted on close (default True). + 'delete' -- whether the file is automatically deleted (default True). + 'delete_on_close' -- if 'delete', whether the file is deleted on close + or on context exit (default True). 'errors' -- the errors argument to io.open (default None) The file is created as mkstemp() would do it. @@ -548,7 +563,7 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, # Setting O_TEMPORARY in the flags causes the OS to delete # the file when it is closed. This is only supported by Windows. - if _os.name == 'nt' and delete: + if _os.name == 'nt' and delete and delete_on_close: flags |= _os.O_TEMPORARY if "b" not in mode: @@ -567,7 +582,7 @@ def opener(*args): raw = getattr(file, 'buffer', file) raw = getattr(raw, 'raw', raw) raw.name = name - return _TemporaryFileWrapper(file, name, delete) + return _TemporaryFileWrapper(file, name, delete, delete_on_close) except: file.close() raise diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index ccf7ea072de276..3746b2e8fd8d2b 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -979,6 +979,65 @@ def test_del_on_close(self): finally: os.rmdir(dir) + def test_not_del_on_close_if_delete_on_close_false(self): + # A NamedTemporaryFile is NOT deleted when closed if + # delete_on_close= False, but is deleted on content manager exit + dir = tempfile.mkdtemp() + try: + with tempfile.NamedTemporaryFile(dir=dir, + delete=True, + delete_on_close=False) as f: + f.write(b'blat') + f_name = f.name + f.close() + with self.subTest(): + # Testing that file is not deleted on close + self.assertTrue(os.path.exists(f.name), + "NamedTemporaryFile %s is incorrectly deleted\ + on closure when delete_on_close= False" % f_name) + + with self.subTest(): + # Testing that file is deleted on content manager exit + self.assertFalse(os.path.exists(f.name), + "NamedTemporaryFile %s exists\ + after content manager exit" % f.name) + + finally: + os.rmdir(dir) + + def test_ok_to_delete_manually(self): + # A NamedTemporaryFile can be deleted by a user before content manager + # comes to it. This will not generate an error + + dir = tempfile.mkdtemp() + try: + with tempfile.NamedTemporaryFile(dir=dir, + delete=True, + delete_on_close=False) as f: + f.write(b'blat') + f.close() + os.unlink(f.name) + + finally: + os.rmdir(dir) + + def test_not_del_if_delete_false(self): + # A NamedTemporaryFile is not deleted if delete = False + dir = tempfile.mkdtemp() + f_name = "" + try: + # setting delete_on_close = True to test, that this does not have + # an effect, if delete = False + with tempfile.NamedTemporaryFile(dir=dir, delete=False, + delete_on_close=True) as f: + f.write(b'blat') + f_name = f.name + self.assertTrue(os.path.exists(f.name), + "NamedTemporaryFile %s exists after close" % f.name) + finally: + os.unlink(f_name) + os.rmdir(dir) + def test_dis_del_on_close(self): # Tests that delete-on-close can be disabled dir = tempfile.mkdtemp() diff --git a/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst b/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst new file mode 100644 index 00000000000000..29f8185b61e3cb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst @@ -0,0 +1 @@ +The :class:`tempfile.NamedTemporaryFile` has a new optional parameter *delete_on_close* From c4cf39f916abd84779e5f59363759308299ed09e Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Thu, 22 Sep 2022 14:57:28 +0200 Subject: [PATCH 02/25] Based on the review from @eryksun the code and the documentation is updated to handle the situation, when delete = True, delete_on_close = Flase and no context manager was used --- Doc/library/tempfile.rst | 8 +++++--- Lib/tempfile.py | 11 +++++++++++ Lib/test/test_tempfile.py | 23 +++++++++++++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 5914731470ff9b..46b8aadc3968c2 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -86,8 +86,8 @@ The module defines the following user-callable items: * There is more granularity in the deletion behaviour of the file (see ``delete_on_close`` below) - The returned object is always a file-like object whose :attr:`!file` - attribute is the underlying true file object. This file-like object can be + The returned object is always a :term:`file-like object` whose :attr:`!file` + attribute is the underlying true file object. This :term:`file-like object` can be used in a :keyword:`with` statement, just like a normal file. The name of the temporary file can be retrieved from the :attr:`name` attribute of the returned file-like object. @@ -95,7 +95,9 @@ The module defines the following user-callable items: If *delete* is true (the default) and *delete_on_close* is true (the default), the file is deleted as soon as it is closed. If *delete* is true and *delete_on_close* is false, the file is deleted on context manager exit - only. If *delete* is false, the value of *delete_on_close* is ignored. + only, if no context manager was used, then the file is deleted only when the + :term:`file-like object` is finalized (see :meth:`object.__del__`). If + *delete* is false, the value of *delete_on_close* is ignored. While the named temporary file is open, the file can always be opened again on POSIX. On Windows, it can be opened again if *delete* is false, or if diff --git a/Lib/tempfile.py b/Lib/tempfile.py index b2b05586902a82..c2a7259d7c83a3 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -530,6 +530,17 @@ def __iter__(self): for line in self.file: yield line + def __del__(self): + # This is to delete the temporary file in case delete = True, + # delete_on_close = False and no context manager was used + if self.delete: + try: + _os.unlink(self.name) + # It is okay to ignore FileNotFoundError. The file may have + # been deleted already. + except FileNotFoundError: + pass + def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 3746b2e8fd8d2b..8d4520039c3aab 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1038,6 +1038,29 @@ def test_not_del_if_delete_false(self): os.unlink(f_name) os.rmdir(dir) + def test_del_by_finalizer_if_no_with(self): + # A NamedTemporaryFile is deleted by fanalizer in case delete = True + # delete_on_close = False and no context manager is used + dir = tempfile.mkdtemp() + try: + f = tempfile.NamedTemporaryFile(dir=dir, delete=True, + delete_on_close=False) + tmp = f.name + f.write(b'blat') + f.close() + with self.subTest(): + self.assertTrue(os.path.exists(tmp), + "NamedTemporaryFile %s missing after close"\ + % tmp) + # Simulating finalizer + f.__del__() + with self.subTest(): + self.assertFalse(os.path.exists(tmp), + "NamedTemporaryFile %s exists after finalizer "\ + % f.name) + finally: + os.rmdir(dir) + def test_dis_del_on_close(self): # Tests that delete-on-close can be disabled dir = tempfile.mkdtemp() From ac0b241572cae7527a5c4fdc19e9a8f16287aad2 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Thu, 22 Sep 2022 15:37:16 +0200 Subject: [PATCH 03/25] documentation is slightly corrected --- Doc/library/tempfile.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 46b8aadc3968c2..6f7617084b42b4 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -96,8 +96,9 @@ The module defines the following user-callable items: default), the file is deleted as soon as it is closed. If *delete* is true and *delete_on_close* is false, the file is deleted on context manager exit only, if no context manager was used, then the file is deleted only when the - :term:`file-like object` is finalized (see :meth:`object.__del__`). If - *delete* is false, the value of *delete_on_close* is ignored. + :term:`file-like object` is finalized and hence deletion is not always + guaranteed (see :meth:`object.__del__`). If *delete* is false, the value of + *delete_on_close* is ignored. While the named temporary file is open, the file can always be opened again on POSIX. On Windows, it can be opened again if *delete* is false, or if From 384614fd09f9dedaf995a35b348710b755916505 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Thu, 22 Sep 2022 17:04:31 +0200 Subject: [PATCH 04/25] minor documentation update --- Doc/library/tempfile.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 6f7617084b42b4..db6c768b352b3e 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -97,8 +97,8 @@ The module defines the following user-callable items: and *delete_on_close* is false, the file is deleted on context manager exit only, if no context manager was used, then the file is deleted only when the :term:`file-like object` is finalized and hence deletion is not always - guaranteed (see :meth:`object.__del__`). If *delete* is false, the value of - *delete_on_close* is ignored. + guaranteed in this case (see :meth:`object.__del__`). If *delete* is false, + the value of *delete_on_close* is ignored. While the named temporary file is open, the file can always be opened again on POSIX. On Windows, it can be opened again if *delete* is false, or if From 0e0b6ba04d62ae6be33abbbc3ed042584908bd4c Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Thu, 22 Sep 2022 18:18:53 +0200 Subject: [PATCH 05/25] Update Doc/whatsnew/3.12.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Éric --- Doc/whatsnew/3.12.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 8f3d494fda64a8..d87696246e697d 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -145,7 +145,7 @@ unicodedata tempfile -------- -The :class:`tempfile.NamedTemporaryFile` class has a new optional parameter +The :class:`tempfile.NamedTemporaryFile` function has a new optional parameter *delete_on_close* (Contributed by Evgeny Zorin in :gh:`58451`.) Optimizations From cc81ab5478d9cc17620a17906c0612bee74339b8 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Fri, 23 Sep 2022 14:40:47 +0200 Subject: [PATCH 06/25] Deletion of file upon finalization is moved from _TemporaryFileWrapper to _TemporaryFileCloser Test test_del_by_finalizer_if_no_with is changed from simulating finalizer by calling __del__() to making sure finalizer really runs --- Lib/tempfile.py | 20 ++++++-------------- Lib/test/test_tempfile.py | 24 ++++++++++++++---------- 2 files changed, 20 insertions(+), 24 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index c2a7259d7c83a3..809743b876114f 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -446,9 +446,6 @@ def close(self, unlink=_os.unlink): if self.delete and self.delete_on_close: unlink(self.name) - # Need to ensure the file is deleted on __del__ - def __del__(self): - self.close() else: def close(self): @@ -456,6 +453,12 @@ def close(self): self.close_called = True self.file.close() + def __del__(self): + self.close() + # Need to ensure the file is deleted on __del__ if delete = True and + # file still exists + if self.delete and _os.path.exists(self.name): + _os.unlink(self.name) class _TemporaryFileWrapper: """Temporary file wrapper @@ -530,17 +533,6 @@ def __iter__(self): for line in self.file: yield line - def __del__(self): - # This is to delete the temporary file in case delete = True, - # delete_on_close = False and no context manager was used - if self.delete: - try: - _os.unlink(self.name) - # It is okay to ignore FileNotFoundError. The file may have - # been deleted already. - except FileNotFoundError: - pass - def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 8d4520039c3aab..2394763cf91a9b 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -11,6 +11,7 @@ import stat import types import weakref +import gc from unittest import mock import unittest @@ -980,6 +981,7 @@ def test_del_on_close(self): os.rmdir(dir) def test_not_del_on_close_if_delete_on_close_false(self): + # Issue gh-58451: tempfile.NamedTemporaryFile is not particulary useful on Windows # A NamedTemporaryFile is NOT deleted when closed if # delete_on_close= False, but is deleted on content manager exit dir = tempfile.mkdtemp() @@ -1008,7 +1010,6 @@ def test_not_del_on_close_if_delete_on_close_false(self): def test_ok_to_delete_manually(self): # A NamedTemporaryFile can be deleted by a user before content manager # comes to it. This will not generate an error - dir = tempfile.mkdtemp() try: with tempfile.NamedTemporaryFile(dir=dir, @@ -1041,23 +1042,26 @@ def test_not_del_if_delete_false(self): def test_del_by_finalizer_if_no_with(self): # A NamedTemporaryFile is deleted by fanalizer in case delete = True # delete_on_close = False and no context manager is used - dir = tempfile.mkdtemp() - try: + def my_func(dir): f = tempfile.NamedTemporaryFile(dir=dir, delete=True, delete_on_close=False) - tmp = f.name + tmp_name = f.name f.write(b'blat') f.close() with self.subTest(): - self.assertTrue(os.path.exists(tmp), + self.assertTrue(os.path.exists(tmp_name), "NamedTemporaryFile %s missing after close"\ - % tmp) - # Simulating finalizer - f.__del__() + % tmp_name) + return tmp_name + # Making sure that Garbage Collector has finalyzed the file object + gc.collect() + dir = tempfile.mkdtemp() + try: + tmp_name = my_func(dir) with self.subTest(): - self.assertFalse(os.path.exists(tmp), + self.assertFalse(os.path.exists(tmp_name), "NamedTemporaryFile %s exists after finalizer "\ - % f.name) + % tmp_name) finally: os.rmdir(dir) From 5af6150aef9911bd4fe2a03b19e25f60ad794f5c Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 00:07:30 +0200 Subject: [PATCH 07/25] Some additions to Lib/test/test_tempfile.py --- Lib/test/test_tempfile.py | 106 +++++++++++++++++++++----------------- 1 file changed, 59 insertions(+), 47 deletions(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 2394763cf91a9b..ad69150fafdede 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -980,8 +980,43 @@ def test_del_on_close(self): finally: os.rmdir(dir) - def test_not_del_on_close_if_delete_on_close_false(self): - # Issue gh-58451: tempfile.NamedTemporaryFile is not particulary useful on Windows + def test_dis_del_on_close(self): + # Tests that delete-on-close can be disabled + dir = tempfile.mkdtemp() + tmp = None + try: + f = tempfile.NamedTemporaryFile(dir=dir, delete=False) + tmp = f.name + f.write(b'blat') + f.close() + self.assertTrue(os.path.exists(f.name), + "NamedTemporaryFile %s missing after close" % f.name) + finally: + if tmp is not None: + os.unlink(tmp) + os.rmdir(dir) + + def test_multiple_close(self): + # A NamedTemporaryFile can be closed many times without error + f = tempfile.NamedTemporaryFile() + f.write(b'abc\n') + f.close() + f.close() + f.close() + + def test_context_manager(self): + # A NamedTemporaryFile can be used as a context manager + with tempfile.NamedTemporaryFile() as f: + self.assertTrue(os.path.exists(f.name)) + self.assertFalse(os.path.exists(f.name)) + def use_closed(): + with f: + pass + self.assertRaises(ValueError, use_closed) + + def test_context_man_not_del_on_close_if_delete_on_close_false(self): + # Issue gh-58451: tempfile.NamedTemporaryFile is not particulary useful + # on Windows # A NamedTemporaryFile is NOT deleted when closed if # delete_on_close= False, but is deleted on content manager exit dir = tempfile.mkdtemp() @@ -1007,7 +1042,7 @@ def test_not_del_on_close_if_delete_on_close_false(self): finally: os.rmdir(dir) - def test_ok_to_delete_manually(self): + def test_context_man_ok_to_delete_manually(self): # A NamedTemporaryFile can be deleted by a user before content manager # comes to it. This will not generate an error dir = tempfile.mkdtemp() @@ -1022,7 +1057,7 @@ def test_ok_to_delete_manually(self): finally: os.rmdir(dir) - def test_not_del_if_delete_false(self): + def test_context_man_not_del_if_delete_false(self): # A NamedTemporaryFile is not deleted if delete = False dir = tempfile.mkdtemp() f_name = "" @@ -1039,65 +1074,42 @@ def test_not_del_if_delete_false(self): os.unlink(f_name) os.rmdir(dir) - def test_del_by_finalizer_if_no_with(self): + def test_del_by_finalizer(self): # A NamedTemporaryFile is deleted by fanalizer in case delete = True # delete_on_close = False and no context manager is used - def my_func(dir): + def my_func(dir)->str: f = tempfile.NamedTemporaryFile(dir=dir, delete=True, delete_on_close=False) tmp_name = f.name f.write(b'blat') - f.close() - with self.subTest(): - self.assertTrue(os.path.exists(tmp_name), - "NamedTemporaryFile %s missing after close"\ - % tmp_name) + # Testing extreme case, where the file is not even explicitly closed + # f.close() return tmp_name - # Making sure that Garbage Collector has finalyzed the file object + # Making sure that Garbage Collector has finalized the file object gc.collect() dir = tempfile.mkdtemp() try: tmp_name = my_func(dir) - with self.subTest(): - self.assertFalse(os.path.exists(tmp_name), - "NamedTemporaryFile %s exists after finalizer "\ - % tmp_name) + self.assertFalse(os.path.exists(tmp_name), + "NamedTemporaryFile %s exists after finalizer "\ + % tmp_name) finally: os.rmdir(dir) - def test_dis_del_on_close(self): - # Tests that delete-on-close can be disabled - dir = tempfile.mkdtemp() - tmp = None - try: - f = tempfile.NamedTemporaryFile(dir=dir, delete=False) - tmp = f.name + def test_correct_finalizer_work_if_already_deleted(self): + # There should be No errors in case delete = True + # delete_on_close = False and no context manager is used, but file is + # deleted manually + def my_func(dir)->str: + f = tempfile.NamedTemporaryFile(dir=dir, delete=True, + delete_on_close=False) + tmp_name = f.name f.write(b'blat') f.close() - self.assertTrue(os.path.exists(f.name), - "NamedTemporaryFile %s missing after close" % f.name) - finally: - if tmp is not None: - os.unlink(tmp) - os.rmdir(dir) - - def test_multiple_close(self): - # A NamedTemporaryFile can be closed many times without error - f = tempfile.NamedTemporaryFile() - f.write(b'abc\n') - f.close() - f.close() - f.close() - - def test_context_manager(self): - # A NamedTemporaryFile can be used as a context manager - with tempfile.NamedTemporaryFile() as f: - self.assertTrue(os.path.exists(f.name)) - self.assertFalse(os.path.exists(f.name)) - def use_closed(): - with f: - pass - self.assertRaises(ValueError, use_closed) + os.unlink(tmp_name) + return tmp_name + # Making sure that Garbage Collector has finalized the file object + gc.collect() def test_bad_mode(self): dir = tempfile.mkdtemp() From 239354e2e4a09914cd9fac0847e9d883c84c0e08 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 00:23:14 +0200 Subject: [PATCH 08/25] clearing of the files in the situation when delete and not delete_on_close is moved to _TemporaryFileWrapper based on the advice from @eryksun There are errors in the test (at least on Linux) --- Lib/tempfile.py | 51 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 16 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 809743b876114f..a091bfeb494c59 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -446,6 +446,9 @@ def close(self, unlink=_os.unlink): if self.delete and self.delete_on_close: unlink(self.name) + # Need to ensure the file is deleted on __del__ + def __del__(self): + self.close() else: def close(self): @@ -453,12 +456,6 @@ def close(self): self.close_called = True self.file.close() - def __del__(self): - self.close() - # Need to ensure the file is deleted on __del__ if delete = True and - # file still exists - if self.delete and _os.path.exists(self.name): - _os.unlink(self.name) class _TemporaryFileWrapper: """Temporary file wrapper @@ -467,6 +464,8 @@ class _TemporaryFileWrapper: temporary use. In particular, it seeks to automatically remove the file when it is no longer needed. """ + name = None + _cleanup_called = False def __init__(self, file, name, delete=True, delete_on_close=True): self.file = file @@ -501,22 +500,31 @@ def __enter__(self): self.file.__enter__() return self + def _cleanup(self): + if self._cleanup_called or self.name is None: + return + self._cleanup_called = True + try: + self.close() + finally: + # If the file is to be deleted, but not on close, delete it now. + if self.delete and not self.delete_on_close: + try: + _os.unlink(self.name) + except FileNotFoundError: + # The file may have been deleted already. + pass + # Need to trap __exit__ as well to ensure the file gets # deleted when used in a with statement def __exit__(self, exc, value, tb): result = self.file.__exit__(exc, value, tb) - self.close() - # If the file is to be deleted, but not on close, delete it now. - if self.delete and not self.delete_on_close: - try: - _os.unlink(self.name) - # It is okay to ignore FileNotFoundError. The file may have - # been deleted already. - except FileNotFoundError: - pass - + self._cleanup() return result + def __del__(self): + self._cleanup() + def close(self): """ Close the temporary file, possibly deleting it. @@ -533,6 +541,17 @@ def __iter__(self): for line in self.file: yield line + def __del__(self): + # This is to delete the temporary file in case delete = True, + # delete_on_close = False and no context manager was used + if self.delete: + try: + _os.unlink(self.name) + # It is okay to ignore FileNotFoundError. The file may have + # been deleted already. + except FileNotFoundError: + pass + def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, From 5de68c0556058c762572e8afe4c2981b38987a1b Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 00:55:13 +0200 Subject: [PATCH 09/25] if statement is added to avoid errors, when testing. Not sure 100% is this is a solution or workaround --- Lib/tempfile.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index a091bfeb494c59..c1814342b9f22b 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -444,7 +444,8 @@ def close(self, unlink=_os.unlink): self.file.close() finally: if self.delete and self.delete_on_close: - unlink(self.name) + if _os.path.exists(self.name): + unlink(self.name) # Need to ensure the file is deleted on __del__ def __del__(self): From eef343984e5d422fd3994fb233a696f313d89927 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 09:59:06 +0200 Subject: [PATCH 10/25] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implemented review from @eryksun and @merwok Co-authored-by: Ezio Melotti Co-authored-by: Eryk Sun Co-authored-by: Éric --- Doc/library/tempfile.rst | 9 ++++---- Lib/tempfile.py | 15 +++---------- Lib/test/test_tempfile.py | 21 +++++++++---------- .../2020-09-28-04-56-04.bpo-14243.YECnxv.rst | 2 +- 4 files changed, 18 insertions(+), 29 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index db6c768b352b3e..6826ceeeac2d98 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -80,11 +80,10 @@ The module defines the following user-callable items: This function operates exactly as :func:`TemporaryFile` does, except the following differences: - * The file is guaranteed to have a visible name in the file system (on Unix, - the directory entry is not unlinked). - - * There is more granularity in the deletion behaviour of the file (see - ``delete_on_close`` below) + * The file is guaranteed to have a visible name in the file system + (on Unix, the directory entry is not unlinked). + * There is more granularity in the deletion behaviour of the file + (see *delete_on_close* below) The returned object is always a :term:`file-like object` whose :attr:`!file` attribute is the underlying true file object. This :term:`file-like object` can be diff --git a/Lib/tempfile.py b/Lib/tempfile.py index c1814342b9f22b..5a576a8ca9f3fc 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -501,7 +501,7 @@ def __enter__(self): self.file.__enter__() return self - def _cleanup(self): + def _cleanup(self, unlink=_os.unlink): if self._cleanup_called or self.name is None: return self._cleanup_called = True @@ -511,7 +511,7 @@ def _cleanup(self): # If the file is to be deleted, but not on close, delete it now. if self.delete and not self.delete_on_close: try: - _os.unlink(self.name) + unlink(self.name) except FileNotFoundError: # The file may have been deleted already. pass @@ -543,16 +543,7 @@ def __iter__(self): yield line def __del__(self): - # This is to delete the temporary file in case delete = True, - # delete_on_close = False and no context manager was used - if self.delete: - try: - _os.unlink(self.name) - # It is okay to ignore FileNotFoundError. The file may have - # been deleted already. - except FileNotFoundError: - pass - + self._cleanup() def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index ad69150fafdede..fafcd8841f2c7c 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1018,7 +1018,7 @@ def test_context_man_not_del_on_close_if_delete_on_close_false(self): # Issue gh-58451: tempfile.NamedTemporaryFile is not particulary useful # on Windows # A NamedTemporaryFile is NOT deleted when closed if - # delete_on_close= False, but is deleted on content manager exit + # delete_on_close=False, but is deleted on context manager exit dir = tempfile.mkdtemp() try: with tempfile.NamedTemporaryFile(dir=dir, @@ -1030,21 +1030,21 @@ def test_context_man_not_del_on_close_if_delete_on_close_false(self): with self.subTest(): # Testing that file is not deleted on close self.assertTrue(os.path.exists(f.name), - "NamedTemporaryFile %s is incorrectly deleted\ - on closure when delete_on_close= False" % f_name) + f"NamedTemporaryFile {f.name!r} is incorrectly deleted " + f"on closure when delete_on_close=False") with self.subTest(): - # Testing that file is deleted on content manager exit + # Testing that file is deleted on context manager exit self.assertFalse(os.path.exists(f.name), - "NamedTemporaryFile %s exists\ - after content manager exit" % f.name) + f"NamedTemporaryFile {f.name!r} exists " + f"after context manager exit") finally: os.rmdir(dir) def test_context_man_ok_to_delete_manually(self): - # A NamedTemporaryFile can be deleted by a user before content manager - # comes to it. This will not generate an error + # A NamedTemporaryFile can be deleted by a user before + # context manager comes to it. This will not generate an error dir = tempfile.mkdtemp() try: with tempfile.NamedTemporaryFile(dir=dir, @@ -1069,7 +1069,7 @@ def test_context_man_not_del_if_delete_false(self): f.write(b'blat') f_name = f.name self.assertTrue(os.path.exists(f.name), - "NamedTemporaryFile %s exists after close" % f.name) + f"NamedTemporaryFile {f.name!r} exists after close") finally: os.unlink(f_name) os.rmdir(dir) @@ -1091,8 +1091,7 @@ def my_func(dir)->str: try: tmp_name = my_func(dir) self.assertFalse(os.path.exists(tmp_name), - "NamedTemporaryFile %s exists after finalizer "\ - % tmp_name) + f"NamedTemporaryFile {tmp_name!r} exists after finalizer ") finally: os.rmdir(dir) diff --git a/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst b/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst index 29f8185b61e3cb..267535452ef146 100644 --- a/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst +++ b/Misc/NEWS.d/next/Library/2020-09-28-04-56-04.bpo-14243.YECnxv.rst @@ -1 +1 @@ -The :class:`tempfile.NamedTemporaryFile` has a new optional parameter *delete_on_close* +The :class:`tempfile.NamedTemporaryFile` function has a new optional parameter *delete_on_close* From c9801eec27f494cd5da99e7604e688adb21fa0be Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 10:20:07 +0200 Subject: [PATCH 11/25] __del__ was defined twice in _TemporaryFileWrapper. Fixed now --- Lib/tempfile.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 5a576a8ca9f3fc..c3288563063f1a 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -542,9 +542,6 @@ def __iter__(self): for line in self.file: yield line - def __del__(self): - self._cleanup() - def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, delete=True, *, errors=None, From 6944ab23e513861508fe643558efce47f178875e Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 11:03:55 +0200 Subject: [PATCH 12/25] _TemporaryFileCloser is returned to original state and the logic, which takes into account delete_on_close moved to _TemporaryFileWrapper (suggestion of @ekisun) --- Lib/tempfile.py | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index c3288563063f1a..627b1c912af40f 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -421,11 +421,10 @@ class _TemporaryFileCloser: file = None # Set here since __del__ checks it close_called = False - def __init__(self, file, name, delete=True, delete_on_close=True): + def __init__(self, file, name, delete=True): self.file = file self.name = name self.delete = delete - self.delete_on_close = delete_on_close # NT provides delete-on-close as a primitive, so we don't need # the wrapper to do anything special. We still use it so that @@ -443,9 +442,8 @@ def close(self, unlink=_os.unlink): try: self.file.close() finally: - if self.delete and self.delete_on_close: - if _os.path.exists(self.name): - unlink(self.name) + if self.delete and _os.path.exists(self.name): + unlink(self.name) # Need to ensure the file is deleted on __del__ def __del__(self): @@ -473,8 +471,8 @@ def __init__(self, file, name, delete=True, delete_on_close=True): self.name = name self.delete = delete self.delete_on_close = delete_on_close - self._closer = _TemporaryFileCloser(file, name, delete, - delete_on_close) + self._closer = _TemporaryFileCloser(file, name, + delete and delete_on_close) def __getattr__(self, name): # Attribute lookups are delegated to the underlying file From 44f0664b3e28e5e0d40ff64238544258b27206ff Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 19:28:20 +0200 Subject: [PATCH 13/25] All delete operations are moved to the class _TemporaryFileCloser modified: Lib/tempfile.py --- Lib/tempfile.py | 78 +++++++++++++++++-------------------------------- 1 file changed, 26 insertions(+), 52 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 627b1c912af40f..5263a8b53d83b0 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -418,44 +418,40 @@ class _TemporaryFileCloser: underlying file object, without adding a __del__ method to the temporary file.""" - file = None # Set here since __del__ checks it + cleanup_called = False close_called = False - def __init__(self, file, name, delete=True): + def __init__(self, file, name, delete=True, delete_on_close=True): self.file = file self.name = name self.delete = delete + self.delete_on_close = delete_on_close - # NT provides delete-on-close as a primitive, so we don't need - # the wrapper to do anything special. We still use it so that - # file.name is useful (i.e. not "(fdopen)") with NamedTemporaryFile. - if _os.name != 'nt': - # Cache the unlinker so we don't get spurious errors at - # shutdown when the module-level "os" is None'd out. Note - # that this must be referenced as self.unlink, because the - # name TemporaryFileWrapper may also get None'd out before - # __del__ is called. - - def close(self, unlink=_os.unlink): - if not self.close_called and self.file is not None: - self.close_called = True - try: + def cleanup(self, windows=(_os.name == 'nt'), unlink=_os.unlink): + if not self.cleanup_called: + self.cleanup_called = True + try: + if not self.close_called: + self.close_called = True self.file.close() - finally: - if self.delete and _os.path.exists(self.name): + finally: + if self.delete and not (self.delete_on_close and windows): + try: unlink(self.name) + except FileNotFoundError: + pass - # Need to ensure the file is deleted on __del__ - def __del__(self): - self.close() - - else: - def close(self): - if not self.close_called: - self.close_called = True + def close(self): + if not self.close_called: + self.close_called = True + try: self.file.close() + finally: + if self.delete and self.delete_on_close: + self.cleanup() - + def __del__(self): + self.cleanup() class _TemporaryFileWrapper: """Temporary file wrapper @@ -463,16 +459,12 @@ class _TemporaryFileWrapper: temporary use. In particular, it seeks to automatically remove the file when it is no longer needed. """ - name = None - _cleanup_called = False def __init__(self, file, name, delete=True, delete_on_close=True): self.file = file self.name = name - self.delete = delete - self.delete_on_close = delete_on_close - self._closer = _TemporaryFileCloser(file, name, - delete and delete_on_close) + self._closer = _TemporaryFileCloser(file, name, delete, + delete_on_close) def __getattr__(self, name): # Attribute lookups are delegated to the underlying file @@ -499,31 +491,13 @@ def __enter__(self): self.file.__enter__() return self - def _cleanup(self, unlink=_os.unlink): - if self._cleanup_called or self.name is None: - return - self._cleanup_called = True - try: - self.close() - finally: - # If the file is to be deleted, but not on close, delete it now. - if self.delete and not self.delete_on_close: - try: - unlink(self.name) - except FileNotFoundError: - # The file may have been deleted already. - pass - # Need to trap __exit__ as well to ensure the file gets # deleted when used in a with statement def __exit__(self, exc, value, tb): result = self.file.__exit__(exc, value, tb) - self._cleanup() + self._closer.cleanup() return result - def __del__(self): - self._cleanup() - def close(self): """ Close the temporary file, possibly deleting it. From 1dfaf72a90f41039ef7786eeac342993f61259da Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 20:12:33 +0200 Subject: [PATCH 14/25] Minor update to comments --- Lib/tempfile.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 5263a8b53d83b0..7cee4d23aa0b45 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -527,7 +527,8 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, 'newline' -- the newline argument to io.open (default None) 'delete' -- whether the file is automatically deleted (default True). 'delete_on_close' -- if 'delete', whether the file is deleted on close - or on context exit (default True). + (default True) or otherwise either on context manager exit + (if context manager was used) or on object finalization. . 'errors' -- the errors argument to io.open (default None) The file is created as mkstemp() would do it. From e54feb5d8621f79ff8437d3209b80b997aa9d199 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 25 Sep 2022 21:33:22 +0200 Subject: [PATCH 15/25] Whitespec fixes by patchcheck.py modified: Lib/tempfile.py --- Lib/tempfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 7cee4d23aa0b45..71b0ec0fd691d4 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -527,7 +527,7 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, 'newline' -- the newline argument to io.open (default None) 'delete' -- whether the file is automatically deleted (default True). 'delete_on_close' -- if 'delete', whether the file is deleted on close - (default True) or otherwise either on context manager exit + (default True) or otherwise either on context manager exit (if context manager was used) or on object finalization. . 'errors' -- the errors argument to io.open (default None) The file is created as mkstemp() would do it. From d2d11194ba0648d6b83b0982cca418911fb2d3ac Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Mon, 26 Sep 2022 10:18:07 +0200 Subject: [PATCH 16/25] Update Lib/tempfile.py Co-authored-by: Eryk Sun --- Lib/tempfile.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 71b0ec0fd691d4..528f6acb543e68 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -452,6 +452,8 @@ def close(self): def __del__(self): self.cleanup() + + class _TemporaryFileWrapper: """Temporary file wrapper From bd4b2ddc10476bc0c817431da34bac1af1f37da5 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Mon, 26 Sep 2022 10:18:25 +0200 Subject: [PATCH 17/25] Update Lib/tempfile.py Co-authored-by: Eryk Sun --- Lib/tempfile.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 528f6acb543e68..3c9d6da4fe60ec 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -435,6 +435,8 @@ def cleanup(self, windows=(_os.name == 'nt'), unlink=_os.unlink): self.close_called = True self.file.close() finally: + # Windows provides delete-on-close as a primitive, in which + # case the file was deleted by self.file.close(). if self.delete and not (self.delete_on_close and windows): try: unlink(self.name) From 29eeaff60ec551fbdd96ea04a202597906334a01 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Mon, 26 Sep 2022 10:42:56 +0200 Subject: [PATCH 18/25] Modified exception handling in NamedTemporaryFile modified: Lib/tempfile.py --- Lib/tempfile.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 3c9d6da4fe60ec..09b134c1c33f72 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -575,7 +575,8 @@ def opener(*args): file.close() raise except: - if name is not None and not (_os.name == 'nt' and delete): + if name is not None and not ( + _os.name == 'nt' and delete and delete_on_close): _os.unlink(name) raise From 5306b6dd52a7813cf8c93d49387106fcae2dadce Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Mon, 26 Sep 2022 16:28:09 +0200 Subject: [PATCH 19/25] Wrapping at 80 characters is fixed modified: Lib/test/test_tempfile.py --- Lib/test/test_tempfile.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index fafcd8841f2c7c..27d3423cef884c 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1030,8 +1030,8 @@ def test_context_man_not_del_on_close_if_delete_on_close_false(self): with self.subTest(): # Testing that file is not deleted on close self.assertTrue(os.path.exists(f.name), - f"NamedTemporaryFile {f.name!r} is incorrectly deleted " - f"on closure when delete_on_close=False") + f"NamedTemporaryFile {f.name!r} is incorrectly " + f"deleted on closure when delete_on_close=False") with self.subTest(): # Testing that file is deleted on context manager exit @@ -1082,7 +1082,7 @@ def my_func(dir)->str: delete_on_close=False) tmp_name = f.name f.write(b'blat') - # Testing extreme case, where the file is not even explicitly closed + # Testing extreme case, where the file is not explicitly closed # f.close() return tmp_name # Making sure that Garbage Collector has finalized the file object @@ -1091,7 +1091,8 @@ def my_func(dir)->str: try: tmp_name = my_func(dir) self.assertFalse(os.path.exists(tmp_name), - f"NamedTemporaryFile {tmp_name!r} exists after finalizer ") + f"NamedTemporaryFile {tmp_name!r} " + f"exists after finalizer ") finally: os.rmdir(dir) @@ -1178,7 +1179,8 @@ def test_iobase_interface(self): missing_attrs = iobase_attrs - spooledtempfile_attrs self.assertFalse( missing_attrs, - 'SpooledTemporaryFile missing attributes from IOBase/BufferedIOBase/TextIOBase' + 'SpooledTemporaryFile missing attributes from ' + 'IOBase/BufferedIOBase/TextIOBase' ) def test_del_on_close(self): From 1c99fb344efe837c139376bb802eeabf971ac7a9 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sat, 1 Oct 2022 17:28:16 +0200 Subject: [PATCH 20/25] Update Lib/tempfile.py Co-authored-by: Eryk Sun --- Lib/tempfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 09b134c1c33f72..bb18d60db0d919 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -437,7 +437,7 @@ def cleanup(self, windows=(_os.name == 'nt'), unlink=_os.unlink): finally: # Windows provides delete-on-close as a primitive, in which # case the file was deleted by self.file.close(). - if self.delete and not (self.delete_on_close and windows): + if self.delete and not (windows and self.delete_on_close): try: unlink(self.name) except FileNotFoundError: From 29b89d3d543e898361af93058b8548f5717682e1 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sat, 1 Oct 2022 17:33:41 +0200 Subject: [PATCH 21/25] Documentation update, based on the feedback from @zooba modified: Doc/library/tempfile.rst --- Doc/library/tempfile.rst | 55 +++++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 6826ceeeac2d98..21d8bfcc66074b 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -94,27 +94,33 @@ The module defines the following user-callable items: If *delete* is true (the default) and *delete_on_close* is true (the default), the file is deleted as soon as it is closed. If *delete* is true and *delete_on_close* is false, the file is deleted on context manager exit - only, if no context manager was used, then the file is deleted only when the - :term:`file-like object` is finalized and hence deletion is not always - guaranteed in this case (see :meth:`object.__del__`). If *delete* is false, - the value of *delete_on_close* is ignored. - - While the named temporary file is open, the file can always be opened again - on POSIX. On Windows, it can be opened again if *delete* is false, or if - *delete_on_close* is false, or if the additional open shares delete access - (e.g. by calling :func:`os.open` with the flag ``O_TEMPORARY``). On - Windows, if *delete* is true and *delete_on_close* is false, additional - opens that do not share delete access (e.g. via builtin :func:`open`) must - be closed before exiting the context manager, else the :func:`os.unlink` - call on context manager exit will fail with a :exc:`PermissionError`. - - To use the name of the temporary file to open the closed file second time, - either make sure not to delete the file upon closure (set the *delete* - parameter to be false) or, in case the temporary file is created in a - :keyword:`with` statement, set the *delete_on_close* parameter to be false. + only, or else when the :term:`file-like object` is finalized. Deletion is not + always guaranteed in this case (see :meth:`object.__del__`). If *delete* is + false, the value of *delete_on_close* is ignored. + + Therefore to use the name of the temporary file to open the closed file + second time, either make sure not to delete the file upon closure (set the + *delete* parameter to be false) or, in case the temporary file is created in + a :keyword:`with` statement, set the *delete_on_close* parameter to be false. The latter approach is recommended as it provides assistance in automatic cleaning of the temporary file upon the context manager exit. + Opening the temporary file again by its name while it is still open works as + follows: + + * On POSIX the file can always be opened again. + * On Windows to open the file again while it is open make sure that at least + one of the following conditions are fulfilled: + + * *delete* is false + * additional open shares delete access (e.g. by calling :func:`os.open` + with the flag ``O_TEMPORARY``) + * *delete* is true but *delete_on_close* is false. Note, that in this + case the additional opens that do not share delete access (e.g. + created via builtin :func:`open`) must be closed before exiting the + context manager, else the :func:`os.unlink` call on context manager + exit will fail with a :exc:`PermissionError`. + On Windows, if *delete_on_close* is false, and the file is created in a directory for which the user lacks delete access, then the :func:`os.unlink` call on exit of the context manager will fail with a :exc:`PermissionError`. @@ -381,19 +387,16 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # file is now closed and removed - # create a temporary file using a context manager, note the name, + # create a temporary file using a context manager # close the file, use the name to open the file again >>> with tempfile.TemporaryFile(delete_on_close=False) as fp: - ... fp.write(b'Hello world!') - ... fp_name = fp.name - ... fp.close() + ... fp.write(b'Hello world!') + ... fp.close() # the file is closed, but not removed # open the file again by using its name - ... f = open(fp_name) - ... f.seek(0) - ... f.read() + ... with open(fp.name) as f + ... f.read() b'Hello world!' - ... f.close() >>> # file is now removed From e58602812bc23388c2eff31234223706cd49693d Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sat, 1 Oct 2022 17:39:17 +0200 Subject: [PATCH 22/25] Minor PEP8 update based on the feedback from @zooba modified: Lib/test/test_tempfile.py --- Lib/test/test_tempfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 27d3423cef884c..dc04ec93baf8b8 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1077,7 +1077,7 @@ def test_context_man_not_del_if_delete_false(self): def test_del_by_finalizer(self): # A NamedTemporaryFile is deleted by fanalizer in case delete = True # delete_on_close = False and no context manager is used - def my_func(dir)->str: + def my_func(dir): f = tempfile.NamedTemporaryFile(dir=dir, delete=True, delete_on_close=False) tmp_name = f.name From 3eceb4549bc0529eddcacec4e0e54bc175d6f28f Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sat, 1 Oct 2022 18:52:03 +0200 Subject: [PATCH 23/25] Improving comments in the init test file, based on the feedback from @eryksun Files changed: Lib/test/test_tempfile.py Co-authored-by: Eryk Sun --- Lib/test/test_tempfile.py | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index dc04ec93baf8b8..7c2c8de7a2e6fc 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1043,8 +1043,8 @@ def test_context_man_not_del_on_close_if_delete_on_close_false(self): os.rmdir(dir) def test_context_man_ok_to_delete_manually(self): - # A NamedTemporaryFile can be deleted by a user before - # context manager comes to it. This will not generate an error + # In the case of delete=True, a NamedTemporaryFile can be manually + # deleted in a with-statement context without causing an error. dir = tempfile.mkdtemp() try: with tempfile.NamedTemporaryFile(dir=dir, @@ -1062,8 +1062,7 @@ def test_context_man_not_del_if_delete_false(self): dir = tempfile.mkdtemp() f_name = "" try: - # setting delete_on_close = True to test, that this does not have - # an effect, if delete = False + # Test that delete_on_close=True has no effect if delete=False. with tempfile.NamedTemporaryFile(dir=dir, delete=False, delete_on_close=True) as f: f.write(b'blat') @@ -1075,8 +1074,8 @@ def test_context_man_not_del_if_delete_false(self): os.rmdir(dir) def test_del_by_finalizer(self): - # A NamedTemporaryFile is deleted by fanalizer in case delete = True - # delete_on_close = False and no context manager is used + # A NamedTemporaryFile is deleted when finalized in the case of + # delete=True, delete_on_close=False, and no with-statement is used. def my_func(dir): f = tempfile.NamedTemporaryFile(dir=dir, delete=True, delete_on_close=False) @@ -1085,7 +1084,7 @@ def my_func(dir): # Testing extreme case, where the file is not explicitly closed # f.close() return tmp_name - # Making sure that Garbage Collector has finalized the file object + # Make sure that the garbage collector has finalized the file object. gc.collect() dir = tempfile.mkdtemp() try: @@ -1097,9 +1096,9 @@ def my_func(dir): os.rmdir(dir) def test_correct_finalizer_work_if_already_deleted(self): - # There should be No errors in case delete = True - # delete_on_close = False and no context manager is used, but file is - # deleted manually + # There should be no error in the case of delete=True, + # delete_on_close=False, no with-statement is used, and the file is + # deleted manually. def my_func(dir)->str: f = tempfile.NamedTemporaryFile(dir=dir, delete=True, delete_on_close=False) @@ -1108,7 +1107,7 @@ def my_func(dir)->str: f.close() os.unlink(tmp_name) return tmp_name - # Making sure that Garbage Collector has finalized the file object + # Make sure that the garbage collector has finalized the file object. gc.collect() def test_bad_mode(self): From 892e87d5f0d9d082a796a0855519624c4f2e8bb4 Mon Sep 17 00:00:00 2001 From: Ev2geny Date: Sun, 2 Oct 2022 19:55:59 +0200 Subject: [PATCH 24/25] Updating of documentation modified: library/tempfile.rst --- Doc/library/tempfile.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 21d8bfcc66074b..15d9ec6025234a 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -80,16 +80,19 @@ The module defines the following user-callable items: This function operates exactly as :func:`TemporaryFile` does, except the following differences: - * The file is guaranteed to have a visible name in the file system - (on Unix, the directory entry is not unlinked). - * There is more granularity in the deletion behaviour of the file - (see *delete_on_close* below) + * This function returns a file that is guaranteed to have a visible name in + the file system. + * To manage the named file, it extends the parameters of + :func:`TemporaryFile` with *delete* and *delete_on_close* parameters that + determine whether and how the named file should be automatically deleted. The returned object is always a :term:`file-like object` whose :attr:`!file` - attribute is the underlying true file object. This :term:`file-like object` can be - used in a :keyword:`with` statement, just like a normal file. The name of the - temporary file can be retrieved from the :attr:`name` attribute of the - returned file-like object. + attribute is the underlying true file object. This :term:`file-like object` + can be used in a :keyword:`with` statement, just like a normal file. The + name of the temporary file can be retrieved from the :attr:`name` attribute + of the returned file-like object. On Unix, unlike with the + :func:`TemporaryFile`, the directory entry does not get unlinked immediately + after the file creation. If *delete* is true (the default) and *delete_on_close* is true (the default), the file is deleted as soon as it is closed. If *delete* is true From 07b963e453a9d15086a819804803b982d7cd94aa Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Tue, 4 Oct 2022 22:12:49 +0100 Subject: [PATCH 25/25] Apply suggestions from code review --- Doc/library/tempfile.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 15d9ec6025234a..b6d4f5dd05bbfc 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -101,8 +101,8 @@ The module defines the following user-callable items: always guaranteed in this case (see :meth:`object.__del__`). If *delete* is false, the value of *delete_on_close* is ignored. - Therefore to use the name of the temporary file to open the closed file - second time, either make sure not to delete the file upon closure (set the + Therefore to use the name of the temporary file to reopen the file after + closing it, either make sure not to delete the file upon closure (set the *delete* parameter to be false) or, in case the temporary file is created in a :keyword:`with` statement, set the *delete_on_close* parameter to be false. The latter approach is recommended as it provides assistance in automatic @@ -112,8 +112,8 @@ The module defines the following user-callable items: follows: * On POSIX the file can always be opened again. - * On Windows to open the file again while it is open make sure that at least - one of the following conditions are fulfilled: + * On Windows, make sure that at least one of the following conditions are + fulfilled: * *delete* is false * additional open shares delete access (e.g. by calling :func:`os.open`