Thanks to visit codestin.com
Credit goes to github.com

Skip to content

os: support creating FIFOs on Windows #103510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RayyanAnsari opened this issue Apr 13, 2023 · 5 comments
Open

os: support creating FIFOs on Windows #103510

RayyanAnsari opened this issue Apr 13, 2023 · 5 comments
Labels
OS-windows type-feature A feature request or enhancement

Comments

@RayyanAnsari
Copy link
Contributor

RayyanAnsari commented Apr 13, 2023

os.mkfifo() creates a named pipe that is accessible on the file system. It is currently only supported on Unix.
Windows also supports creating named pipes with CreateNamedPipeW() which returns a handle to the server end of the pipe. However, they can only be created on a special filesystem in the \\.\pipe\ directory.

The Windows docs say that:

An instance of a named pipe is always deleted when the last handle to the instance of the named pipe is closed.

And os.mkfifo doesn't return the pipe handles, it just creates the pipe:

Note that mkfifo() doesn’t open the FIFO — it just creates the rendezvous point.

This would mean that implementing Windows support into os.mkfifo would mean that the pipe is deleted as soon as it is created (if we close the handle returned by CreateNamedPipeW()), or we have a pipe that we can't delete (if we leave the pipe handle open).
Unless there is some way to pass that back to the calling code - like os.pipe() that's supported on Windows.

I'm not sure how I would go about implementing this. Returning the handle only on Windows? Making a new cross-platform function that always returns the handle (perhaps called os.fifo)?

Linked PRs

@RayyanAnsari RayyanAnsari added the type-feature A feature request or enhancement label Apr 13, 2023
@eryksun
Copy link
Contributor

eryksun commented Apr 13, 2023

The semantics are significantly different for a FIFO on POSIX compared to a named pipe on Windows.

On POSIX, a FIFO is half-duplex, and it opens as a single file in the kernel that has a write end and a read end. Either end of the FIFO can be opened by name multiple times, depending on whether open() is called with O_RDONLY, O_WRONLY, or O_RDWR.

On Windows, a named pipe is opened as a pipe instance in the named-pipe filesystem ('\\.\PIPE\"), which connects two kernel file objects -- one for each end of the pipe, the server side and the client side. The number of allowed instances of a pipe can be limited up to 254, or unlimited. The client end can be remote (over SMB) if the pipe mode isn't restricted to local clients. A named pipe can be inbound (server reads; client writes), outbound (server writes; client reads), or duplex. For the latter, the server end reads what the client end writes, and the client end reads what the server end writes. Unlike a POSIX FIFO, a named pipe can be opened by name only once on each end of an instance of the pipe -- once by the server end via CreateNamePipeW() and once on the client end via CreateFileW(). That said, multiple handles for the kernel file object on either end of a pipe instance may be duplicated to or inherited by other processes.

@SuibianP
Copy link

What about this -

On Windows, pipe servers are created by os.mkfifo, and pipe clients are created by normal open. os.mkfifo takes a new keyword argument open_mode=None, which causes the function to return an open fd corresponding to the mode if it is not None. The pipe will always be created in full duplex mode for now, but it should be trivial to extend another kwarg for pipe_mode if the need arises.

FIFO_PATH = "/tmp/myfifo"
os.mkfifo(FIFO_PATH)
with open(FIFO_PATH, "w") as mypipe:
    mypipe.write("Message\n")

becomes

FIFO_PATH = "/tmp/myfifo"
pipe_fd = os.mkfifo(FIFO_PATH, open_flags=os.WRONLY)
with os.fdopen(pipe_fd, "w") as mypipe:
    mypipe.write("Message\n")

The deletion semantics will be different between Unix and Windows, although I don't think it matters as Windows pipes cannot be persistent. Moreover, the API will of course only present identical semantics for the case of one single read & write end. Opening a named pipe multiple times will by nature exhibit different semantics between Unix and Windows.

Comments? @eryksun

SuibianP added a commit to SuibianP/cpython that referenced this issue Jan 28, 2025
Make os.mkfifo return an open fd on Windows and the original path object
on Unix. The user would then pass the return value into open, which
accepts both path and fd as file.
SuibianP added a commit to SuibianP/cpython that referenced this issue Jan 28, 2025
Make os.mkfifo return an open fd on Windows and the original path object
on Unix. The user would then pass the return value into open, which
accepts both path and fd as file.
@2trvl
Copy link
Contributor

2trvl commented Feb 4, 2025

@SuibianP From Eryk Sun's comment and my thoughts, it can be seen that Unix FIFO and Windows Named Pipes are very different. Mimicking Unix FIFO is difficult.

So I propose creating a new function os.namedpipe(). See updated description.

The code will look like this:

# name just string, default mode "x+", raise an OSError if pipe exists, FILE_FLAG_FIRST_PIPE_INSTANCE
mypipe_fd = os.namedpipe(name, mode)
# since 3.12 available as SetNamedPipeHandleState(PIPE_NOWAIT)
os.set_blocking(mypipe_fd, False)
# create a new mypipe instance
mypipe2_fd = os.namedpipe(name, "w+")

Duplex Modes:

  • x+, create PIPE_ACCESS_DUPLEX
  • r+, open existing
  • w+, open new instance

Server Write, Client Read Modes:

  • x, create PIPE_ACCESS_OUTBOUND
  • r, open existing with read only
  • w, open new instance

New Instance Exclusive Modes:

  • a+, open new PIPE_ACCESS_DUPLEX instance (equivalent to w+)
  • a, open new PIPE_ACCESS_OUTBOUND instance (equivalent to w)

@SuibianP
Copy link

SuibianP commented Feb 4, 2025

(Continuing the discussion from #129420)

I dislike option 2, since existing code using mode would suddenly leaks a file descriptor whereas the code works fine with Python 3.13. I suggest to continue the discussion in the issue instead.

You may add a new function instead.

@vstinner open_mode was to be a new keyword argument, distinct from the existing octal mode argument. As such, there was to be no change of behaviour for existing code.


@2trvl

(Sorry for not replying in one batch; still grokking your proposals)

The simplest case will be os.mkfifo, which allows only 2 processes, but this is not very cool.

This is exactly what I meant by "simple IPC case", which is the scope of common semantics I intended to support. IMHO this would cover the majority of named pipe use cases, and is self-contained without opening a can of worms as you noted. I considered platform-specific features (message-based FIFOs, MPMC semantics) out of scope for my original proposal.

Frankly, I am not sure if anything beyond that would be very useful with a semantic difference this great -- a quick search did not find real-world Python code using os.mkfifo or pywin32 win32pipe.CreateNamedPipe in multi-producer or multi-consumer ways. Would it be better if we first implement the building blocks by adding new keyword arguments into mkfifo, and add the higher level abstraction only when there is a more established usage pattern in the wild?

@2trvl
Copy link
Contributor

2trvl commented Feb 6, 2025

@SuibianP It is preferable to use existing Python file calls rather than creating new ones.

os.namedpipe(npfs: str, mode: str = "w+", wait: bool = True, open: bool = False) -> int

Creates a new Windows named pipe instance. The function's behavior is something between os.mkfifo and os.pipe. It creates the rendezvous point and opens a server end.

Args:

  • npfs - Named pipe file system path in win32 device namespace \\.\pipe\name.
  • mode - Create mode. Must be the same for all instances. Defaults to "w+".
  • wait - Wait for client end connection. Ensures clear synchronization. When combined with os.set_blocking(fd, wait) it can make a named pipe nonblocking. Defaults to True.
  • open - Open named pipe instance for reading and writing by Everyone. This can be used to communicate with processes of other users or remote clients over SMB. Defaults to False.

Returns:

  • int - Named pipe server end. The new file descriptor is non-inheritable.

Raises:

  • OSError - Unknown or invalid create mode. Access denied.

Available create modes:

Mode Access Internal
w+ read & write PIPE_ACCESS_DUPLEX
w write PIPE_ACCESS_OUTBOUND
r read PIPE_ACCESS_INBOUND

Local nonblocking write:

mypipe_fd = os.namedpipe(r"\\.\pipe\mypipe", wait=False)
os.set_blocking(mypipe_fd, False)
mypipe = os.fdopen(mypipe_fd, "w")
mypipe.write("Hello from mypipe server!")
mypipe = open(r"\\.\pipe\mypipe")
print(mypipe.read())
# Hello from mypipe server!

Remote:

mypipe_fd = os.namedpipe(r"\\.\pipe\mypipe", open=True)
mypipe = os.fdopen(mypipe_fd, "w+")
mypipe = open(r"\\ComputerName\pipe\mypipe", "w+")
mypipe.write("Message over SMB")

Microsoft Named Pipe Operations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OS-windows type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants