gh-103510: Support `os.mkfifo` on Windows #129420

SuibianP · 2025-01-28T20:49:04Z

Happy Year of the Snake!

The current common practice of os.mkfifo then open the path cannot possibly be made to work on Windows (barring global state hacks at least), due to the named pipe semantics and indivisibility at NtCreateNamedPipeFile syscall level. That said, it should be feasible to support os.mkfifo on Windows for the simple IPC case (named pipe with single R/W end) with certain API changes. There seems to be two ways forward:

os.mkfifo to return an open fd on Windows and the original path object on Unix. The user would then pass the return value into open, which accepts both path and fd as file. The fd, however, will always be O_RDWR on Windows.
os.mkfifo to accept an additional open_mode argument, and always returns an open fd or file object on any platform.

This experimental patch that would fix #103510 implements the simpler former for a start.

It would have been convenient to automatically prepend the \\.\pipe\ prefix so that os.mkfifo("/tmp/mypipe") can be truly platform independent, but this is not currently the case on VxWorks and the user likely would still need the full filename. Maybe something like os.pipe_prefix would help.

Any comments would be much appreciated.

Issue: os: support creating FIFOs on Windows #103510

ghost · 2025-01-28T20:49:07Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-01-28T20:49:11Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Make os.mkfifo return an open fd on Windows and the original path object on Unix. The user would then pass the return value into open, which accepts both path and fd as file.

bedevere-app · 2025-01-28T20:50:42Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Wulian233 · 2025-01-29T13:38:43Z

Happy Year of the Snake! Could you add news entry(see bot) and update the document description?

https://github.com/python/cpython/blob/main/Doc%2Flibrary%2Fos.rst#L2586-L2586

It would be even better if it could add whatnew3.14. Thanks!

SuibianP · 2025-01-29T16:04:34Z

Hi @Wulian233, thanks for the response! For this draft PR I have not yet polished the formalities (and test cases as well), since the exact choice of API change is still unsettled. Once there is consensus (1 vs 2, whether to restrict number of instances, etc.), I will complete those and mark this PR ready for review.

vstinner · 2025-01-30T15:30:41Z

Modules/posixmodule.c

+    Py_END_ALLOW_THREADS
+    if (INVALID_HANDLE_VALUE == h)
+        return win32_error_object("CreateNamedPipeW", path->object);
+    fd = _Py_open_osfhandle(h, _O_RDWR);


_Py_open_osfhandle() can return -1: see the Windows implementation of os.pipe().

vstinner

It's strange that that Windows implementation returns a file descriptor, whereas the Unix implementation returns None. I'm not sure if this change is a good idea :-(

vstinner · 2025-01-30T15:32:22Z

Modules/posixmodule.c

@@ -12544,6 +12544,21 @@ static PyObject *
 os_mkfifo_impl(PyObject *module, path_t *path, int mode, int dir_fd)
 /*[clinic end generated code: output=ce41cfad0e68c940 input=73032e98a36e0e19]*/
 {
+#ifdef MS_WINDOWS


You should raise an error if dir_fd is used.

You should document that the mode is ignored on Windows.

bedevere-app · 2025-01-30T15:33:29Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

SuibianP · 2025-01-30T16:08:44Z

It's strange that that Windows implementation returns a file descriptor, whereas the Unix implementation returns None. I'm not sure if this change is a good idea :-(

Would option 2 be better in this case? It would be more verbose, but could avoid the multiplexed return type.

vstinner · 2025-01-30T16:39:48Z

os.mkfifo to accept #103510 (comment), and always returns an open fd or file object on any platform.

This sounds like an incompatible change. You cannot simply modify os.mkfifo() to return a file descriptor. It sounds like a new function to me.

SuibianP · 2025-01-30T16:53:24Z

Sorry I did not make it clear, old code that does not specify open_mode would continue to have the old semantics and return None on Unix and will raise on Windows.

2trvl · 2025-02-03T22:50:44Z

@SuibianP It's not as easy as you think.

First, don't add any new arguments. Use the mode argument as in os.mkdir to restrict pipe access to the current user and administrator. Passing dir_fd should raise an OSError.

I suggest you create a symbolic link to \\.\pipe\path-hash in place of path. You'll have to save the created HANDLEs somewhere so they don't close immediately. Then os.open and io.open will use them if necessary.

Because a named pipe can only be opened once on each side, you need to create the illusion of the opposite. With os.mkfifo you call CreateNamedPipe(\\.\pipe\path-hash-r, PIPE_ACCESS_INBOUND) and CreateNamedPipe(\\.\pipe\path-hash-w, PIPE_ACCESS_OUTBOUND, FILE_FLAG_OVERLAPPED). That is, you create 2 named pipes for different operations.

Create 2 new objects from io.IOBase: NamedPipeServer and NamedPipeClient. NamedPipeServer will be returned by io.open(path) within the process that called os.mkfifo(path), but only once. os.mkfifo must check pipe existence with FILE_FLAG_FIRST_PIPE_INSTANCE. NamedPipeClient is used when a named pipe exists.

NamedPipeServer.read :
ConnectNamedPipe(\.\pipe\path-hash-w, overlapped)
ReadFile(\.\pipe\path-hash-w, overlapped)
DisconnectNamedPipe(\.\pipe\path-hash-w)
sleep, next if no timeout on overlapped

NamedPipeServer.write :
ConnectNamedPipe(\.\pipe\path-hash-r)
WriteFile(\.\pipe\path-hash-r)
DisconnectNamedPipe(\.\pipe\path-hash-r)

NamedPipeClient.read :
CreateFile(\.\pipe\path-hash-r)
ReadFile(\.\pipe\path-hash-r)

NamedPipeClient.write :
CreateFile(\.\pipe\path-hash-w)
WriteFile(\.\pipe\path-hash-w)

All operations will be blocking, everything written will be received by the process that started it. Overlapped is required to read all incoming messages from several processes. The read data is not duplicated and is transferred to the first client.

To create a named pipe identical to Unix, you will need a separate thread that will create a new NamedPipe instance for each open.

Thread Main Loop :
CreateNamedPipe(hash-w, PIPE_ACCESS_OUTBOUND, FILE_FLAG_OVERLAPPED)
ConnectNamedPipe(hash-w, overlapped); if failed, save for next iteration, don't create new
for instances: ReadFile(hash-w, overlapped)
ConnectNamedPipe(hash-r, overlapped), if success, WriteFile(hash-r, all-read-data), DisconnectNamedPipe(hash-r)

You also need to consider whether open() will be a blocking operation. Should you add os.O_NONBLOCK exclusively for Windows named pipes?

Examples from Microsoft documentation

2trvl · 2025-02-03T23:28:00Z

I think we could consider changing os.mkfifo return value.

os.mkfifo to return an open fd on Windows and the original path object on Unix.

Then it would close the issue with:

save the created HANDLEs somewhere so they don't close immediately

And also allow open() to distinguish objects like:
fd -> NamedPipeServer
path -> NamedPipeClient

Dividing into Server and Client objects only allows creating several Clients. And a separate thread will constantly consume resources and slow down the interpreter.

The simplest case will be os.mkfifo, which allows only 2 processes, but this is not very cool.

vstinner · 2025-02-04T00:15:11Z

It's strange that that Windows implementation returns a file descriptor, whereas the Unix implementation returns None. I'm not sure if this change is a good idea :-(

I close the issue. You should go back to the design phase and first propose an API which works on all platforms. IMO modifying os.mkfifo() to return a file descriptor only on Windows is not an option.

SuibianP · 2025-02-04T10:24:28Z

@vstinner

It's strange that that Windows implementation returns a file descriptor, whereas the Unix implementation returns None. I'm not sure if this change is a good idea :-(

I close the issue. You should go back to the design phase and first propose an API which works on all platforms. IMO modifying os.mkfifo() to return a file descriptor only on Windows is not an option.

Please kindly take a look at API option 2 as proposed in my original PR body and second response, where os.mkfifo would always return an fd on all platforms if open_mode is specified.

vstinner · 2025-02-04T10:41:37Z

I dislike option 2, since existing code using mode would suddenly leaks a file descriptor whereas the code works fine with Python 3.13. I suggest to continue the discussion in the issue instead.

You may add a new function instead.

2trvl · 2025-02-04T10:44:13Z

@SuibianP take a look at this

bedevere-app bot mentioned this pull request Jan 28, 2025

os: support creating FIFOs on Windows #103510

Open

pythongh-103510: Support os.mkfifo on Windows

3cafd46

Make os.mkfifo return an open fd on Windows and the original path object on Unix. The user would then pass the return value into open, which accepts both path and fd as file.

SuibianP force-pushed the os-mkfifo-windows branch from 7fb0442 to 3cafd46 Compare January 28, 2025 20:50

SuibianP changed the title ~~gh-103510: Support os.mkfifo on Windows~~ gh-103510: Support os.mkfifo on Windows Jan 28, 2025

vstinner reviewed Jan 30, 2025

View reviewed changes

vstinner requested changes Jan 30, 2025

View reviewed changes

bedevere-app bot added the awaiting changes label Jan 30, 2025

vstinner closed this Feb 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-103510: Support `os.mkfifo` on Windows #129420

gh-103510: Support `os.mkfifo` on Windows #129420

SuibianP commented Jan 28, 2025 •

edited by bedevere-app bot

Loading

ghost commented Jan 28, 2025 •

edited by ghost

Loading

bedevere-app bot commented Jan 28, 2025

bedevere-app bot commented Jan 28, 2025

Wulian233 commented Jan 29, 2025 •

edited

Loading

SuibianP commented Jan 29, 2025

vstinner Jan 30, 2025

vstinner left a comment

vstinner Jan 30, 2025

bedevere-app bot commented Jan 30, 2025

SuibianP commented Jan 30, 2025

vstinner commented Jan 30, 2025

SuibianP commented Jan 30, 2025 •

edited

Loading

2trvl commented Feb 3, 2025 •

edited

Loading

2trvl commented Feb 3, 2025 •

edited

Loading

vstinner commented Feb 4, 2025

SuibianP commented Feb 4, 2025

vstinner commented Feb 4, 2025 •

edited

Loading

2trvl commented Feb 4, 2025

gh-103510: Support os.mkfifo on Windows #129420

gh-103510: Support os.mkfifo on Windows #129420

Conversation

SuibianP commented Jan 28, 2025 • edited by bedevere-app bot Loading

ghost commented Jan 28, 2025 • edited by ghost Loading

bedevere-app bot commented Jan 28, 2025

bedevere-app bot commented Jan 28, 2025

Wulian233 commented Jan 29, 2025 • edited Loading

SuibianP commented Jan 29, 2025

vstinner Jan 30, 2025

Choose a reason for hiding this comment

vstinner left a comment

Choose a reason for hiding this comment

vstinner Jan 30, 2025

Choose a reason for hiding this comment

bedevere-app bot commented Jan 30, 2025

SuibianP commented Jan 30, 2025

vstinner commented Jan 30, 2025

SuibianP commented Jan 30, 2025 • edited Loading

2trvl commented Feb 3, 2025 • edited Loading

2trvl commented Feb 3, 2025 • edited Loading

vstinner commented Feb 4, 2025

SuibianP commented Feb 4, 2025

vstinner commented Feb 4, 2025 • edited Loading

2trvl commented Feb 4, 2025

gh-103510: Support `os.mkfifo` on Windows #129420

gh-103510: Support `os.mkfifo` on Windows #129420

SuibianP commented Jan 28, 2025 •

edited by bedevere-app bot

Loading

ghost commented Jan 28, 2025 •

edited by ghost

Loading

Wulian233 commented Jan 29, 2025 •

edited

Loading

SuibianP commented Jan 30, 2025 •

edited

Loading

2trvl commented Feb 3, 2025 •

edited

Loading

2trvl commented Feb 3, 2025 •

edited

Loading

vstinner commented Feb 4, 2025 •

edited

Loading