bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)

vstinner · web-flow · commit e6f8abd50075 · 2020-04-24T12:06:58.000+02:00
Optimize the subprocess module on FreeBSD using closefrom(). A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX) is high, the loop calling close(fd) on each file descriptor can take several milliseconds. The workaround on FreeBSD to improve performance was to load and mount the fdescfs kernel module, but this is not enabled by default. Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
@@ -471,6 +471,10 @@ Optimizations
   until the main thread handles signals.
   (Contributed by Victor Stinner in :issue:`40010`.)
 
+* Optimize the :mod:`subprocess` module on FreeBSD using ``closefrom()``.
+  (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor
+  Stinner in :issue:`38061`.)
+
 
 Build and C API Changes
 =======================
diff --git a/Misc/NEWS.d/next/Library/2020-04-24-01-55-00.bpo-38061.XmULB3.rst b/Misc/NEWS.d/next/Library/2020-04-24-01-55-00.bpo-38061.XmULB3.rst
@@ -0,0 +1,11 @@
+Optimize the :mod:`subprocess` module on FreeBSD using ``closefrom()``.
+A single ``close(fd)`` syscall is cheap, but when ``sysconf(_SC_OPEN_MAX)`` is
+high, the loop calling ``close(fd)`` on each file descriptor can take several
+milliseconds.
+
+The workaround on FreeBSD to improve performance was to load and mount the
+fdescfs kernel module, but this is not enabled by default.
+
+Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and
+Kubilay Kocak (koobs):
+https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c
@@ -264,9 +264,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_fds_to_keep)
         start_fd = keep_fd + 1;
     }
     if (start_fd <= end_fd) {
+#if defined(__FreeBSD__)
+        /* Any errors encountered while closing file descriptors are ignored */
+        closefrom(start_fd);
+#else
         for (fd_num = start_fd; fd_num < end_fd; ++fd_num) {
-            close(fd_num);
+            /* Ignore errors */
+            (void)close(fd_num);
         }
+#endif
     }
 }
 

Original file line number	Diff line number	Diff line change
`@@ -264,9 +264,15 @@ _close_fds_by_brute_force(long start_fd, PyObject *py_fds_to_keep)`
`264`	`264`	`start_fd = keep_fd + 1;`
`265`	`265`	`}`
`266`	`266`	`if (start_fd <= end_fd) {`
	`267`	`+#if defined(__FreeBSD__)`
	`268`	`+ /* Any errors encountered while closing file descriptors are ignored */`
	`269`	`+ closefrom(start_fd);`
	`270`	`+#else`
`267`	`271`	`for (fd_num = start_fd; fd_num < end_fd; ++fd_num) {`
`268`		`- close(fd_num);`
	`272`	`+ /* Ignore errors */`
	`273`	`+ (void)close(fd_num);`
`269`	`274`	`}`
	`275`	`+#endif`
`270`	`276`	`}`
`271`	`277`	`}`
`272`	`278`