Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-87414: add musl support to platform.libc_ver #103784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from

Conversation

cmaureir
Copy link
Contributor

@cmaureir cmaureir commented Apr 24, 2023

This includes an implementation from the packaging module to detect the musl version on the system,
allowing libc_ver to properly return the version rather than an empty tuple.

This includes an implementation from the packaging
module to detect the musl version on the system,
allowing libc_ver to properly return the version
rather than an empty tuple.

Co-authored-by: Manuel Kaufmann <[email protected]>
@humitos
Copy link
Contributor

humitos commented Apr 24, 2023

We QAed this patch locally and we got it working. However, we are not 100% sure if this is a good implementation and, in the case it is, we are not sure if the place were we copied the functions and put the new file is OK, so:

  • Is it fine adding the _elffile.py in the Lib/ directory?
  • Are tests for this class required? Where we should add them? Should we add a binary file to the repository so execute the tests over? 🤔
  • We copy some of the functions from packaging._musllinux.py into Lib/platform.py. Is that OK or we should create a Lib/_musllinux.py instead?
  • We are checking for b'musl' in binary inside the while loop and the opening the same binary again to be parsed by ELFFile. The idea behind this was to not open the executable twice for all the cases, but only for musl. We are not convinced about this either, but we are not sure what could be better.

cc @pradyunsg @malemburg

@malemburg
Copy link
Member

A few comments:

  • Please add tests for this (possibly mocking things using dumps of actual musl files)
  • Please don't add type annotations to the platform stdlib code. If you want to add those, please post them to typesched.
  • Adding the helper module to Lib/ is fine, just please add a notice to the module that this is an internal module and not meant for public consumption.
  • Before adding the helper module, please make sure you get proper license information for it from the original source and add this to the module doc-string.
  • Function doc-strings always go first in a function definition, not after some imports (see _get_musl_version()). Please adjust the code.
  • Please replace the NamedTuple _MuslVersion with a simple Python tuple. No need to over-engineer the code.

I have no way to test the code, so can only do source level review. Please find someone who can test this in real life.

Thanks.

@cmaureir
Copy link
Contributor Author

Thank you Marc 🎉

I addressed the following points:

Please don't add type annotations to the platform stdlib code. If you want to add those, please post them to typesched.

Adding the helper module to Lib/ is fine, just please add a notice to the module that this is an internal module and not meant for public consumption.

Function doc-strings always go first in a function definition, not after some imports (see _get_musl_version()). Please adjust the code.

Please replace the NamedTuple _MuslVersion with a simple Python tuple. No need to over-engineer the code.

Before adding the helper module, please make sure you get proper license information for it from the original source and add this to the module doc-string.

I naively assumed that due to the code being used inside the packaging module it should have been straightforward, but I totally understand, will be double check :)

We will work on this soon with @humitos

Please add tests for this (possibly mocking things using dumps of actual musl files)

I have no way to test the code, so can only do source level review. Please find someone who can test this in real life.

Understood, thanks! maybe the person who reported it can give us a hand.

ambv and others added 2 commits April 25, 2023 02:09
We get the following error otherwise:

```
ImportError: attempted relative import with no known parent package
```
@humitos
Copy link
Contributor

humitos commented Apr 25, 2023

I have no way to test the code, so can only do source level review. Please find someone who can test this in real life.

This is how we are testing this code. I hope it makes sense:

  1. Pull the Alpine Docker image and run it
docker pull alpine
docker run -it -u root -v ~/Source/cpython:/mnt/cpython sh
  1. Compile Python inside Alpine
export PYTHON_BUILD_PACKAGES="\
      bzip2-dev \
      coreutils \
      dpkg \
      dpkg-dev \
      expat-dev \
      findutils \
      gcc \
      gdbm-dev \
      libc-dev \
      libffi-dev \
      libnsl-dev \
      libtirpc-dev \
      linux-headers \
      make \
      ncurses-dev \
      libressl-dev \
      pax-utils \
      readline-dev \
      sqlite-dev \
      tcl-dev \
      tk \
      tk-dev \
      util-linux-dev \
      xz-dev \
      zlib-dev \
    "

apk add --no-cache --virtual .build-deps $PYTHON_BUILD_PACKAGES
cd /mnt/cpython
./configure
make
  1. Execute Python and check the response
Python 3.12.0a7+ (main, Apr 25 2023, 16:58:19) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.libc_ver()
('musl', '1.2')
>>> 

Note: based on https://github.com/jfloff/alpine-python/blob/master/3.8/Dockerfile#L33

@malemburg
Copy link
Member

Thanks for the instructions, @humitos. I forgot that Docker containers are an easy way to access Alpine builds :-)

BTW: I thought a bit more about adding a new modules to Lib/ and would like to revert my earlier comment. For the small extra logic that is needed, it will create more maintenance burden than is worth it. Could you please try to add the required code directly the platform module ? Perhaps it can be refactored and shortened, since you are not really using all of the parsing logic.

Thanks.

@humitos
Copy link
Contributor

humitos commented Apr 25, 2023

I updated the PR to live only in the platform.py module. I also trimmed the code to remove the things that are not strictly required for this purpose.

I wrote some small tests for ELFFile class and two tests for the functions that uses it. These are under a @support.requires_musl decorator that I created to avoid running these tests in Windows, Mac and Linux versions that don't have musl installed.

Please, let us know if there are more changes required 😄

Copy link
Member

@malemburg malemburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still a few minor things to fix, but overall, this looks good.

Please also add a NEWS entry.

Thanks.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@malemburg
Copy link
Member

The test fails on Macs due to a missing "ldd". On Macs, you typically use "otool" instead.

It would be better to simply return False in requires_musl() or use "otool" instead on "ldd" on Macs.

@mara004
Copy link
Contributor

mara004 commented Nov 27, 2023

@humitos Thanks for working on this PR. May I ask how this is going on? Seems to be stalled with requested changes since May...

@blubberdiblub
Copy link

@cmaureir this pull request seems nearly ready for merging, only 3 small issues (comment for the musl in binary check in platform.py, fixing the requires_musl() function to work gracefully on MacOS and replacing the web links into pypa/packaging sources with permalinks pinned to a specific commit/tag) remain which should be easy to address.

Would be really nice if this could be merged soon

- Making requires_musl compatible with macOS
- Add permalinks on the packaging references
- Add comment for the b'musl' check before b'libc'
@cmaureir
Copy link
Contributor Author

Super sorry @malemburg @blubberdiblub I thought this was merged long time ago 😢
I will keep an eye on this issue until it gets in 👍 thanks for your time.

@support.requires_subprocess()
@support.requires_musl()
def test_libc_ver_musl(self):
self.assertEqual(platform.libc_ver(), ("musl", "1.2"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this always be 1.2?

@encukou
Copy link
Member

encukou commented Mar 8, 2024

@malemburg, if you want to merge this, please go ahaed. I'll write about what would make me comfortable with merging it.


This is quite a lot of code for an unsupported platform.
By unsupported I mean both that:

  • musl is not only not officially listed in PEP 11, and
  • we don't have musl in CI; while we have a buildbot I doubt anyone's checking it; there's no person to contact with musl-related issues.

Getting a platform supported is a bit of catch-22: you need a core dev, and it's hard to become one if you're interested in an unsupported platform. To help solve that, I'd like to figure out how community support for a platform could work.

Is anyone here interested in maintaining CPython on musl? Ideally someone interested in eventually becoming a core dev and bringing musl to official Tier 3 support? If so, please add yourself to the experts index with a PR here.

The indication that someone wants to help keep CPython working on musl, with Marc-Andre's “this looks good” comment, would convince me to merge this.

@malemburg
Copy link
Member

This is quite a lot of code for an unsupported platform. By unsupported I mean both that:

This is really only a side effect of adding ELF header parsing instead of using the file output. We may be able to make good use of it on other platforms as well.

Regarding supporting the musl platform: I think we simply have to accept that CPython will run on unsupported platforms as well 🙂

Alpine is a very widely used platform for running CPython on Docker/Podman, whether we have official tier-X support for it or not. Simply rejecting patches for such a common platform, just because we can't check all the boxes for official tier-X support feels somewhat unreasonable -- at least to me.

Of course, happy to learn if someone wants to help check those boxes for Alpine platforms.

@mara004
Copy link
Contributor

mara004 commented Mar 8, 2024

This is quite a lot of code for an unsupported platform.

Given that there has been a successful PEP for musllinux platform tags (656), "unsupported" sounds a bit hyperbolic to me. 😅 Python is definitely being used on musl systems.

there's no person to contact with musl-related issues

Not in CPython itself perhaps, but there are still downstream maintainers -- Alpine packagers would probably be happy to help.

@encukou
Copy link
Member

encukou commented Mar 11, 2024

This is really only a side effect of adding ELF header parsing instead of using the file output.

Sure. Then again, the first thing I'd like to ask a musl maintainer is whether ELF parsing is the proper way to get the libc version.
(If it breaks somewhere in CPython's 5-year support period, it won't be easy to fix. Maybe users would be better off using a third-party platform identification library, which would be easier to update?)

Not in CPython itself perhaps, but there are still downstream maintainers -- Alpine packagers would probably be happy to help.

Yup. We don't test it here, so we can't really take responsibility for fixing it when it breaks. And we don't have musl experts on board. That's an issue to be fixed, and while I can't fix it myself, I can help on the organizational side of things.
I suppose Alpine packagers do run the test suite and carry necessary patches. I'm all for inviting them to do some of that work in this repository, if they want to sign up for it.

@zware
Copy link
Member

zware commented Apr 28, 2025

As near as I can tell, this has mostly been superseded by GH-131313. Is there anything here that can/should improve on what was done there?

@malemburg
Copy link
Member

Indeed and the solution there is much leaner than this one. I'm going to close it for now.

Thanks @cmaureir for investing the time. Please reopen the issue if you think there are still some parts which would be useful to have in the platform module.

@mcepl
Copy link
Contributor

mcepl commented Apr 29, 2025

docker pull alpine
docker run -it -u root -v ~/Source/cpython:/mnt/cpython sh

export PYTHON_BUILD_PACKAGES="\
      bzip2-dev \
      coreutils \
      dpkg \
      dpkg-dev \
      expat-dev \
      findutils \
      gcc \
      gdbm-dev \
      libc-dev \
      libffi-dev \
      libnsl-dev \
      libtirpc-dev \
      linux-headers \
      make \
      ncurses-dev \
      libressl-dev \
      pax-utils \
      readline-dev \
      sqlite-dev \
      tcl-dev \
      tk \
      tk-dev \
      util-linux-dev \
      xz-dev \
      zlib-dev \
    "

apk add --no-cache --virtual .build-deps $PYTHON_BUILD_PACKAGES
cd /mnt/cpython
./configure
make

For anybody who tries that, let me add that Python officially doesn’t support LibreSSL anymore (e.g., #89532), replace libressl-dev with openssl-dev.

And yes, platform.libc_ver() works:

/mnt/cpython # ./python
Python 3.14.0a7+ (main, Apr 29 2025, 18:48:02) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.freedesktop_os_release()
{'NAME': 'Alpine Linux', 'ID': 'alpine', 'PRETTY_NAME': 'Alpine Linux v3.21', 'VERSION_ID': '3.21.3', 'HOME_URL': 'https://alpinelinux.org/', 'BUG_REPORT_URL': 'https://gitlab.alpinelinux.org/alpine/aports/-/issues'}
>>> platform.libc_ver()
('musl', '1.2.5')
>>> 
/mnt/cpython # 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.