GH-80789: Bundle `ensurepip` wheels at build time #109130

AA-Turner · 2023-09-08T08:47:45Z

Based on the discussion in #12791, this is a sketch of a different approach (building very heavily on @webknjaz's work).

We add a fairly simple bundler script at Tools/build/bundle_ensurepip_wheels.py, with two changes to ensurepip itself to adapt. This PR doesn't attempt to switch to the one-project model, as I found the diff was too large to reasonably review.

One question: should we make this part of the default build process (i.e. integration into make all / PCBuild)?

A

Issue: Replace bundled pip and setuptools with a downloader in the ensurepip module #80789

📚 Documentation preview 📚: https://cpython-previews--109130.org.readthedocs.build/

Co-authored-by: Sviatoslav Sydorenko <[email protected]>

This reverts commit 0aaf135.

AA-Turner · 2023-09-08T11:02:00Z

0aaf135 was an attempt to integrate this into the Makefile/PCBuild more 'properly', but I'm not confident in making wider-scale changes here (said commit didn't work, so I've reverted it).

zware

This is going to need something for buildbots to avoid breaking them all, but I like the direction here.

Doc/whatsnew/3.13.rst

Tools/build/bundle_ensurepip_wheels.py

ned-deily · 2023-09-08T18:28:30Z

Adding @ambv and @Yhg1s for review of impact on release process and supply chain security.

# Conflicts: # .cirrus.yml

hugovk · 2023-09-10T13:22:38Z

.cirrus.yml

@@ -20,6 +20,8 @@ freebsd_task:
  pythoninfo_script:
    - cd build
    - make pythoninfo
+  bundle_ensurepip_script:


Suggested change

bundle_ensurepip_script:

# Download wheels for the venv step

bundle_ensurepip_script:

Is this the reason for adding it here?

Anything using venv or ensurepip during testing needs this provisioning step. If there's no uses, it'd be unnecessary.

The actual reason to include it is that Cirrus CI failed without this and passed with it. I don't know why Free BSD requires the venv step though, as all other tests pass normally -- only documentation and hypothesis (which use venvs) seem to have an explicit venv step.

A

@AA-Turner It's not because of venv but ensurepip itself. The call chain looks as follows:

test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script calls a CPython build preparation helper @ https://github.com/python/cpython/blob/cbb3a6f/Lib/test/test_tools/test_freeze.py#L28

That, in turn, calls make install here https://github.com/python/cpython/blob/cbb3a6f/Tools/freeze/test/freeze.py#L184.

make install calls ensurepip at https://github.com/python/cpython/blob/cbb3a6f/Makefile.pre.in#L1932-L1933 to provision it into the new Python install directory.

ensurepip uses the bundled pip wheel to run bootstrapping so it attempts to unpack it and freaks out when there's no dist for it: https://github.com/python/cpython/blob/cbb3a6f/Lib/ensurepip/__init__.py#L176.

why Free BSD requires

It's not that FreeBSD or Cirrus requires the test explicitly, it's rather that this test is skipped on most platforms via https://github.com/python/cpython/blob/cbb3a6f/Lib/test/test_tools/test_freeze.py#L11-L20.

Lib/test/test_tools/test_bundle_ensurepip_wheels.py

Reviewed-by: Hugo van Kemenade <[email protected]>

webknjaz · 2023-09-10T15:20:14Z

This is going to need something for buildbots to avoid breaking them all, but I like the direction here.

I identified the place needing to be changed in #12791 (comment).

@AA-Turner FYI

AA-Turner · 2023-09-10T15:25:24Z

I identified the place needing to be changed

I tried to explore a different approach of fully integrating this into make all in 0aaf135, which I think may be a better approach as it doesn't require downsteam distributors to change anything in their processes. I defer to those with much more experience with the build systems, though.

A

webknjaz · 2023-09-10T22:05:35Z

I tried to explore a different approach of fully integrating this into make all

I tried doing that in my PR and was getting weird behavior, I wasn't sure why. Maybe I was misusing something in the build machinery or was just confused...

If that works, it's probably fine. OTOH, you may want to take into account that many distributions prefer their build envs to be disconnected from the internet so it might still be wise to provide them with means to pre-provision the wheel separately.

Tagging @encukou @hroncok @mgorny for downstream opinions.

webknjaz · 2023-09-10T23:41:39Z

This PR doesn't attempt to switch to the one-project model, as I found the diff was too large to reasonably review.

Would #109245 help?

mgorny · 2023-09-11T02:24:09Z

If that works, it's probably fine. OTOH, you may want to take into account that many distributions prefer their build envs to be disconnected from the internet so it might still be wise to provide them with means to pre-provision the wheel separately.

In Gentoo we aren't actually using the wheels originally bundled with CPython itself but providing the newest versions of the them separately, so as long as that continues working, I suppose that's fine with us. However, it is is paramount that:

The build process and test suite fully respect --with-wheel-pkg-dir and do not attempt to download anything.
This is entirely restricted to CPython build-time, and in particularly creating venv doesn't attempt any fetching.

However, I can imagine it could be mildly annoying to users who aren't used to the new workflow that having a complete git clone would no longer suffice to actually build CPython offline.

hroncok · 2023-09-11T06:15:50Z

Same for Fedora. With an addition that we do sometimes keep the bundled wheels for too new or too old Pythons, so having a way to pre-populate the wheels instead of downloaidng them would still be needed.

pradyunsg · 2023-10-04T20:19:20Z

@AA-Turner could you resolve the conflicts with this PR?

I think it would be good to have this be in Python 3.13's cycle as early as possible, to allow any potential redistributors who care about the details of ensurepip to have time to adjust things as well as provide us feedback on this as early as feasible.

webknjaz · 2023-10-11T23:10:29Z

Tools/build/bundle_ensurepip_wheels.py

+    spec = importlib.util.spec_from_file_location("ensurepip", ENSURE_PIP_INIT)
+    ensurepip = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(ensurepip)
+    return ensurepip._PROJECTS


@AA-Turner I believe this would need to be updated after #109245, and it'll probably let you work with simpler structures.

webknjaz · 2023-10-11T23:11:50Z

Tools/build/bundle_ensurepip_wheels.py

+                whl = response.read()
+        except URLError as exc:
+            print_error(f"Failed to download {wheel_url!r}: {exc}")
+            errors = 1


Was this meant to be a counter?

Suggested change

errors = 1

errors += 1

webknjaz · 2023-10-11T23:13:15Z

Tools/build/bundle_ensurepip_wheels.py

+                continue
+            else:
+                print_error(f"An invalid '{name}' wheel exists.")
+                os.remove(wheel_path)


Any reason not to use pathlib's method? Seeing that it's already used everywhere..

webknjaz · 2023-10-11T23:16:30Z

Tools/build/bundle_ensurepip_wheels.py

+from urllib.error import URLError
+from urllib.request import urlopen
+
+HOST = 'https://files.pythonhosted.org'


Can we name this something like a PYPI_CDN_URL for maintainability?

webknjaz · 2023-10-11T23:17:09Z

Tools/build/bundle_ensurepip_wheels.py

+
+def print_notice(message: str) -> None:
+    if GITHUB_ACTIONS:
+        print(f"::notice::{message}", end="\n\n")


Nit: this also works if output to stderr FYI.

webknjaz · 2023-10-11T23:20:13Z

Tools/build/bundle_ensurepip_wheels.py

+
+    try:
+        projects = _get_projects()
+    except (AttributeError, TypeError):


I think there's an abstraction leak here: it's hard to guess from looking at the _get_projects() function, what in it might trigger these exceptions. I'd recommend processing them inside that function and making obvious such places where the exceptions may occur. Instead, I'd convert both of the exceptions into something like an ImportError and handle just that. This would contribute to transparency of how that function works.

webknjaz · 2023-10-11T23:23:11Z

Tools/build/bundle_ensurepip_wheels.py

+        return 1
+
+    errors = 0
+    for name, version, checksum in projects:


FWIW with #109245, looping will stop making sense here. So maybe you could start working with just projects[0] already?

No, let's keep them separate.

I was kinda assuming that the simplification PR would get merged first, in which case, this one would have to adapt..

webknjaz · 2023-10-11T23:25:58Z

Doc/whatsnew/3.13.rst

@@ -907,6 +907,10 @@ Build Changes
 * Building CPython now requires a compiler with support for the C11 atomic
  library, GCC built-in atomic functions, or MSVC interlocked intrinsics.

+* Wheels for :mod:`ensurepip` are no longer bundled in the CPython source
+  tree. Distributors should bundle these as part of the build process by
+  running :file:`Tools/build/bundle_ensurepip_wheels.py`.


This would probably be clearer:

Suggested change

running :file:`Tools/build/bundle_ensurepip_wheels.py`.

running :file:`Tools/build/bundle_ensurepip_wheels.py` with no arguments.

webknjaz · 2023-10-11T23:34:23Z

Lib/test/test_ensurepip.py

+        # ensure the pip wheel exists
+        pip_filename = os.path.join(test.support.STDLIB_DIR, 'ensurepip', '_bundled',
+                                    f'pip-{ensurepip._PIP_VERSION}-py3-none-any.whl')
+        if not os.path.isfile(pip_filename):


I feel like if it didn't exist, there should be some code to clean it up after testing so that the dummy file doesn't get forgotten on disk. Tests should avoid side effects...

webknjaz · 2023-10-11T23:35:56Z

Lib/test/test_tools/test_bundle_ensurepip_wheels.py

+    import bundle_ensurepip_wheels as bew
+
+
+# Disable fancy GitHub actions output during the tests


There should be a test for enabled GHA, just for those helpers, then. It'd be sad not to get coverage there.

webknjaz · 2023-10-12T09:03:58Z

Tools/build/bundle_ensurepip_wheels.py

+
+
+def _is_valid_wheel(content: bytes, *, checksum: str) -> bool:
+    return checksum == sha256(content, usedforsecurity=False).hexdigest()


@AA-Turner I already know that this effort is going to be rejected for now, but out of curiosity — what's the motivation for setting usedforsecurity=False? The other verification script doesn't have that: https://github.com/python/cpython/blob/3d18034/Tools/build/verify_ensurepip_wheels.py#L81.

pradyunsg · 2023-10-12T11:47:34Z

Closing this per #80789 (comment). Thanks @AA-Turner for filing this, even though we're not going ahead with this (despite me implying that earlier in the discussion).

webknjaz and others added 4 commits September 7, 2023 00:32

Remove bundled pip wheel

0d8c94f

Add bundle_ensurepip_wheels

c2c9a55

Co-authored-by: Sviatoslav Sydorenko <[email protected]>

Remove verify_ensurepip_wheels

76de0b8

Co-authored-by: Sviatoslav Sydorenko <[email protected]>

Add Makefile target

252ee8e

AA-Turner requested review from pfmoore, pradyunsg and zware September 8, 2023 08:47

AA-Turner requested review from ezio-melotti and hugovk as code owners September 8, 2023 08:47

bedevere-bot added the awaiting review label Sep 8, 2023

bedevere-bot mentioned this pull request Sep 8, 2023

Replace bundled pip and setuptools with a downloader in the ensurepip module #80789

Closed

pfmoore approved these changes Sep 8, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Sep 8, 2023

AA-Turner added 3 commits September 8, 2023 11:05

Merge branch 'main' into ensurepip/bundler-tools

800c487

Attempt to fix Cirrus CI

4f0b3a2

Integrate into build

0aaf135

AA-Turner requested a review from a team as a code owner September 8, 2023 10:56

Revert "Integrate into build"

a163396

This reverts commit 0aaf135.

zware reviewed Sep 8, 2023

View reviewed changes

Doc/whatsnew/3.13.rst Outdated Show resolved Hide resolved

pradyunsg reviewed Sep 8, 2023

View reviewed changes

Tools/build/bundle_ensurepip_wheels.py Outdated Show resolved Hide resolved

Changes from first review

ccaad2d

AA-Turner mentioned this pull request Sep 9, 2023

Only comment about new commits if the PR is still open. python/bedevere#583

Open

Merge branch 'main' into ensurepip/bundler-tools

db6f0db

# Conflicts: # .cirrus.yml

pradyunsg approved these changes Sep 9, 2023

View reviewed changes

hugovk reviewed Sep 10, 2023

View reviewed changes

Add an explanatory comment to Cirrus CI

8fae12a

Reviewed-by: Hugo van Kemenade <[email protected]>

Merge branch 'main' into ensurepip/bundler-tools

161f304

webknjaz reviewed Oct 11, 2023

View reviewed changes

webknjaz reviewed Oct 12, 2023

View reviewed changes

pradyunsg closed this Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-80789: Bundle `ensurepip` wheels at build time #109130

GH-80789: Bundle `ensurepip` wheels at build time #109130

AA-Turner commented Sep 8, 2023 •

edited by github-actions bot

Loading

AA-Turner commented Sep 8, 2023

zware left a comment

ned-deily commented Sep 8, 2023

hugovk Sep 10, 2023

webknjaz Sep 10, 2023

AA-Turner Sep 10, 2023

webknjaz Sep 10, 2023

webknjaz Sep 10, 2023

webknjaz commented Sep 10, 2023

AA-Turner commented Sep 10, 2023

webknjaz commented Sep 10, 2023

webknjaz commented Sep 10, 2023

mgorny commented Sep 11, 2023

hroncok commented Sep 11, 2023

pradyunsg commented Oct 4, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

pradyunsg Oct 12, 2023

webknjaz Oct 12, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 11, 2023

webknjaz Oct 12, 2023

pradyunsg commented Oct 12, 2023 •

edited

Loading

	bundle_ensurepip_script:
	# Download wheels for the venv step
	bundle_ensurepip_script:

	running :file:`Tools/build/bundle_ensurepip_wheels.py`.
	running :file:`Tools/build/bundle_ensurepip_wheels.py` with no arguments.

		import bundle_ensurepip_wheels as bew


		# Disable fancy GitHub actions output during the tests



		def _is_valid_wheel(content: bytes, *, checksum: str) -> bool:
		return checksum == sha256(content, usedforsecurity=False).hexdigest()

GH-80789: Bundle ensurepip wheels at build time #109130

GH-80789: Bundle ensurepip wheels at build time #109130

Conversation

AA-Turner commented Sep 8, 2023 • edited by github-actions bot Loading

AA-Turner commented Sep 8, 2023

zware left a comment

Choose a reason for hiding this comment

ned-deily commented Sep 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Sep 10, 2023

AA-Turner commented Sep 10, 2023

webknjaz commented Sep 10, 2023

webknjaz commented Sep 10, 2023

mgorny commented Sep 11, 2023

hroncok commented Sep 11, 2023

pradyunsg commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg commented Oct 12, 2023 • edited Loading

GH-80789: Bundle `ensurepip` wheels at build time #109130

GH-80789: Bundle `ensurepip` wheels at build time #109130

AA-Turner commented Sep 8, 2023 •

edited by github-actions bot

Loading

pradyunsg commented Oct 12, 2023 •

edited

Loading