GH-87804: Fix counter overflow in statvfs on macOS #99570

ronaldoussoren · 2022-11-18T10:02:15Z

On macOS the statvfs interface returns block counts as 32-bit integers, and that results in bad reporting for larger disks.

Therefore reimplement statvfs in terms of statfs, which does use 64-bit integers for block counts.

Tested using a sparse filesystem image of 100TB.

Issue: MacOS os.statvfs() has rollover for >4TB disks at each 4TB (32bit counter overflow?) #87804

On macOS the statvfs interface returns block counts as 32-bit integers, and that results in bad reporting for larger disks. Therefore reimplement statvfs in terms of statfs, which does use 64-bit integers for block counts. Tested using a sparse filesystem image of 100TB.

ronaldoussoren · 2022-11-18T10:09:04Z

The conversion from struct statfs to a statvfs value is based on the implementation in Apple's libc, in particular for the reported maximum filename length.

I'm not entirely sure if how far back we should back port this. The patch is clean so could easily be back ported to 3.10, but is conceptually a fairly large change.

giampaolo

Patch LGTM. The only thing which I'm doubtful about is the f_flag field. According to the man pages these are the original statvfs fields:

ST_MANDLOCK             Mandatory locking is permitted on the filesystem (see fcntl(2)).
ST_NOATIME              Do not update access times; see mount(2).
ST_NODEV                Disallow access to device special files on this filesystem.
ST_NODIRATIME           Do not update directory access times; see mount(2).
ST_NOEXEC               Execution of programs is disallowed on this filesystem.
ST_NOSUID               The set-user-ID and set-group-ID bits are ignored by exec(3) for executable files on this filesystem
ST_RDONLY               This filesystem is mounted read-only.
ST_RELATIME             Update atime relative to mtime/ctime; see mount(2).
ST_SYNCHRONOUS          Writes are synched to the filesystem immediately (see the description of O_SYNC in open(2)).

...and these are macOS fstatfs's counterparts:

MNT_RDONLY             A read-only filesystem
MNT_SYNCHRONOUS        File system is written to synchronously
MNT_NOEXEC             Can't exec from filesystem
MNT_NOSUID             Setuid bits are not honored on this filesystem
MNT_NODEV              Don't interpret special files
MNT_UNION              Union with underlying filesysten
MNT_ASYNC              File system written to asynchronously
MNT_EXPORTED           File system is exported File system is stored locally
MNT_QUOTA              Quotas are enabled on this file system
MNT_ROOTFS             This file system is the root of the file system
MNT_DOVOLFS            File system supports volfs
MNT_DONTBROWSE         File system is not appropriate path to user data
MNT_UNKNOWNPERMISSIONS VFS will ignore ownership information on filesys-tem filesystem tem objects
MNT_AUTOMOUNTED        File system was mounted by automounter
MNT_JOURNALED          File system is journaled
MNT_DEFWRITE           File system should defer writes
MNT_MULTILABEL         MAC support for individual labels

In your PR you rightly map MNT_RDONLY and MNT_NOSUID to ST_RDONLY and ST_NOSUID (note: you can also map MNT_NOEXEC to ST_NOEXEC) , but the other ST_* flags are lost. I'm not sure if this is appropriate, but perhaps it would make sense to call both statvfs and fstatfs, and use the former to fetch all flags.

giampaolo · 2022-11-28T11:15:01Z

Modules/posixmodule.c

+   }
+   if (st.f_flags & MNT_NOSUID) {
+       flags |= ST_NOSUID;
+   }


In addition to these, I think you can also map MNT_NOEXEC to ST_NOEXEC.

My current implementation matches that of Apple's emulation, but I agree that it is useful to add a mapping for *_NOEXEC as well.

Safihre · 2023-12-01T14:54:39Z

@ronaldoussoren Do you think you might revisit this? :)

ronaldoussoren · 2023-12-23T21:49:44Z

@ronaldoussoren Do you think you might revisit this? :)

I am revisiting this. Finally...

ronaldoussoren · 2023-12-25T09:37:36Z

In your PR you rightly map MNT_RDONLY and MNT_NOSUID to ST_RDONLY and ST_NOSUID (note: you can also map MNT_NOEXEC to ST_NOEXEC) , but the other ST_* flags are lost. I'm not sure if this is appropriate, but perhaps it would make sense to call both statvfs and fstatfs, and use the former to fetch all flags.

The statvfs implementation on macOS: https://github.com/apple-oss-distributions/Libc/blob/c5a3293354e22262702a3add5b2dfc9bb0b93b85/emulated/statvfs.c#L35

As you can see this implementation does not implement flags other than ST_RDONLY and ST_NOSUID. ~~Adding ST_NOEXEC is easy enough, as is ST_NODEV~~. But calling statvfs(3) to get other flags doesn't give us more information.

The current manual page, and SDK headers, only list two flags:

     There are two flags defined for the f_flag member:

           ST_RDONLY      The file system is mounted read-only.

           ST_NOSUID      The semantics of the S_ISUID and S_ISGID file mode bits are not supported by, or are disabled on, this file system.

That's another reason to not call statvfs: There are no other flags to expose than the ones we already emulate.

ronaldoussoren · 2023-12-29T13:32:00Z

@Yhg1s and @pablogsal : What's your opinion on back porting this to 3.12 and 3.11?

The PR reimplements os.statvfs on macOS, but should result in the same behaviour other than fixing the problem with large disks because I based it on Apple implementation of statvfs(3).

ronaldoussoren · 2024-02-10T09:35:56Z

@Yhg1s and @pablogsal : What's your opinion on back porting this to 3.12 and 3.11?

The PR reimplements os.statvfs on macOS, but should result in the same behaviour other than fixing the problem with large disks because I based it on Apple implementation of statvfs(3).

I won't backport to 3.11 and 3.12, the change is too large for that (IMHO)

sobolevn · 2024-02-10T10:36:55Z

Follow up: #115236

bedevere-bot · 2024-02-10T10:53:37Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 6e222a5.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/146/builds/7668) and take a look at the build logs.
Check if the failure is related to this commit (6e222a5) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/146/builds/7668

Failed tests:

test.test_concurrent_futures.test_shutdown

Failed subtests:

test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_concurrent_futures\test_shutdown.py", line 50, in test_interpreter_shutdown
    self.assertEqual(out.strip(), b"apple")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'' != b'apple'

On macOS the statvfs interface returns block counts as 32-bit integers, and that results in bad reporting for larger disks. Therefore reimplement statvfs in terms of statfs, which does use 64-bit integers for block counts. Tested using a sparse filesystem image of 100TB.

ronaldoussoren requested a review from a team November 18, 2022 10:02

bedevere-bot mentioned this pull request Nov 18, 2022

MacOS os.statvfs() has rollover for >4TB disks at each 4TB (32bit counter overflow?) #87804

Closed

bedevere-bot added the awaiting core review label Nov 18, 2022

📜🤖 Added by blurb_it.

8b0c0e5

giampaolo mentioned this pull request Nov 18, 2022

[macOS] Inaccurate info in psutil.disk_usage giampaolo/psutil#2174

Open

Merge branch 'main' into gh87804-statvfs-32bit

57a6013

giampaolo reviewed Nov 28, 2022

View reviewed changes

ronaldoussoren added 2 commits December 22, 2023 11:27

Merge branch 'main' into gh87804-statvfs-32bit

fc98b52

Fix linting issue

4eae818

Merge branch 'main' into gh87804-statvfs-32bit

25e1919

Merge branch 'main' into gh87804-statvfs-32bit

44c2201

ronaldoussoren merged commit 6e222a5 into python:main Feb 10, 2024

bedevere-app bot removed the awaiting core review label Feb 10, 2024

ronaldoussoren deleted the gh87804-statvfs-32bit branch February 10, 2024 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

GH-87804: Fix counter overflow in statvfs on macOS #99570

GH-87804: Fix counter overflow in statvfs on macOS #99570

Uh oh!

ronaldoussoren commented Nov 18, 2022 •

edited by bedevere-bot

Loading

Uh oh!

ronaldoussoren commented Nov 18, 2022

Uh oh!

giampaolo left a comment •

edited

Loading

Uh oh!

giampaolo Nov 28, 2022

Uh oh!

ronaldoussoren Nov 28, 2022

Uh oh!

Safihre commented Dec 1, 2023

Uh oh!

ronaldoussoren commented Dec 23, 2023

Uh oh!

ronaldoussoren commented Dec 25, 2023 •

edited

Loading

Uh oh!

ronaldoussoren commented Dec 29, 2023

Uh oh!

ronaldoussoren commented Feb 10, 2024

Uh oh!

sobolevn commented Feb 10, 2024

Uh oh!

bedevere-bot commented Feb 10, 2024

Uh oh!

Uh oh!

Uh oh!

GH-87804: Fix counter overflow in statvfs on macOS #99570

GH-87804: Fix counter overflow in statvfs on macOS #99570

Uh oh!

Conversation

ronaldoussoren commented Nov 18, 2022 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ronaldoussoren commented Nov 18, 2022

Uh oh!

giampaolo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giampaolo Nov 28, 2022

Choose a reason for hiding this comment

Uh oh!

ronaldoussoren Nov 28, 2022

Choose a reason for hiding this comment

Uh oh!

Safihre commented Dec 1, 2023

Uh oh!

ronaldoussoren commented Dec 23, 2023

Uh oh!

ronaldoussoren commented Dec 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ronaldoussoren commented Dec 29, 2023

Uh oh!

ronaldoussoren commented Feb 10, 2024

Uh oh!

sobolevn commented Feb 10, 2024

Uh oh!

bedevere-bot commented Feb 10, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

ronaldoussoren commented Nov 18, 2022 •

edited by bedevere-bot

Loading

giampaolo left a comment •

edited

Loading

ronaldoussoren commented Dec 25, 2023 •

edited

Loading