Symfony version(s) affected

4.4, 5.4, 6.3

Description

When cache is mounted on NFS with full locking support available, the cache locking fails. Every request will begin to generate and build the container in a large race until one succeeds and successfully writes the cache file.

NFS with full locking support available refers to NFSv3 with recent kernels and rpc.statd running, and NFSv4 which natively supports it. Both support it through fcntl F_SETLK or F_SETLKW. Relevant flock calls are converted. So it always becomes advisory.

I'm aware there's lots of information around on lots of sites about PHP flock and NFS not compatible but it's all very old information and extremely out of date. In summary, it is compatible, and does indeed work as I believe the implementers intended, and has done for a very long while. It's just this one quirk on the file handle open mode requirements when it comes to NFS. Hopefully this issue will provide useful in providing a full outline of the support and how it works, as I put in the context all the main areas involved.

Thanks! Love to Symfony's Team ❤️

How to reproduce

Setup a Symfony framework skeleton and put it on an NFSv4 mount. Run the bin/console several times on different machines mounted to that NFS. You'll see multiple container builds appear in the cache folder. (There is a single PHP entry for the cache - e.g. appAppKernelProdContainer.php - but it loads a container folder with a random name - e.g. ContainerFUJetMW - so you can see when it doesn't lock as it'll have multiple ContainerXXXXXXX in place.)

Performing strace shows that LOCK_EX | LOCK_NB flock call is successful for the first cache build. The next that races in results in a EAGAIN ($wouldBlock is then set to true). It then attempts to acquire a read lock with LOCK_SH but that subsequently fails as the flock calls flock system call and the NFS client translates to fcntl which results in EBADF as per the documentation for fcntl because the original fopen did not include read mode, only write mode.

Possible Solution

Open the lock file with w+.

Confirmed to resolve the problem completely.

Additional Context

Code:

https://github.com/symfony/http-kernel/blob/6.3/Kernel.php#L430-L431

man nfs on most OS shows following support - this is from a CentOS 7 machine:

The Network Lock Manager protocol is a separate sideband protocol used to manage file locks in NFS version 2 and version 3. To support lock recovery after a client or server reboot, a second sideband protocol -- known as the Network Status Manager protocol -- is also required. In NFS version 4, file locking is supported directly in the main NFS protocol, and the NLM and NSM sideband protocols are not used.
[...]
NLM supports advisory file locks only. To lock NFS files, use fcntl(2) with the F_GETLK and F_SETLK commands. The NFS client converts file locks obtained via flock(2) to advisory locks.

https://linux.die.net/man/2/fcntl

In order to place a read lock, fd must be open for reading. In order to place a write lock, fd must be open for writing. To place both types of lock, open a file read-write.
[...]
EBADF
fd is not an open file descriptor, or the command was F_SETLK or F_SETLKW and the file descriptor open mode doesn't match with the type of lock requested.

PHP is using flock under the hood for flock for BC reasons - fcntl is added via other means I think an extension. So Symfony is using flock which calls flock system call and NFS client translates to fcntl.