fix(pypi): pull fewer wheels with experimental_index_url #3058

aignas · 2025-07-05T03:20:44Z

Before this we would pull all of the wheels that the user target
configuration would be compatible with and that meant that it was not
customizable. This also meant that there were a lot of footguns in the
configuration where the select statements were not really foolproof.

With this PR we select only those sources that need to be for the
declared configurations.

Freethreaded support should be done by defining extra freethreaded
platforms using the new builder API. It is done as a followup in #3063.

This is also changing the default platforms to be only the fully supported
platforms. This makes the testing easier and avoids us running into
compatibility issues during the rollout.

Work towards #2747
Fixes #2759
Fixes #2849

aignas · 2025-07-05T03:22:23Z

Created this as WIP to signal that I am working on this and to invite feedback. The CI should be mostly happy but there are a few TODOs before we can fully merge and I want to test it a little bit more.

Freethreaded platform definition should be possible.
Probably something else that I need to think of more carefully.

TODO cleanup after this PR is proven to not break things:

Remove complex config_setting generation where we are generating them from whl filenames - it is just too complex to properly maintain and it has not proven to be effective here. In chore(pypi): remove unused config setting code #3065

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T12:46:43Z

The CI failure is a flake.

This splits the configuration that we have for the `rules_python` and the defaults that we set for our users from the actual unit tests where we check that the extension is working correctly. With this we will be able to dog food the API and point users into the `MODULE.bazel` as the example snippet. Work towards bazel-contrib#2949 Work towards bazel-contrib#2747

Before it seems that we were relying on particular names in the pipstar platforms. This ensures that we rely on this less. Whilst at it fix a few typos and improve the formatting of the code. Work towards bazel-contrib#2949 Work towards bazel-contrib#2747

Before this we would pull all of the wheels that the user target configuration would be compatible with and that meant that it was not customizable. This also meant that there were a lot of footguns in the configuration where the select statements were not really foolproof. With this PR we select only those sources that need to be for the declared configurations. Freethreaded support should be done by defining extre freethreaded platforms using the new builder API. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T14:12:17Z

This would be ready to be reviewed, but because it is stacked on two PRs, let's get those out of the way first.

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.

MODULE.bazel

examples/bzlmod/MODULE.bazel

python/private/pypi/extension.bzl

rickeylev · 2025-07-06T17:46:14Z

python/private/pypi/extension.bzl

+        )) | values.env
+        platforms[key] = struct(
+            env = env_,
+            want_abis = [


What's the difference between the abi computed above, and the want_abis here?

python/private/pypi/extension.bzl

python/private/pypi/select_whl.bzl

keith · 2025-07-07T17:01:14Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
  <builtin>: in <toplevel>
Repository rule python_repository defined at:
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
   Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

aignas · 2025-07-07T23:48:06Z

Thank you, I'll take a look later.

Marking this as a draft for now.

aignas · 2025-07-08T04:10:17Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
  <builtin>: in <toplevel>
Repository rule python_repository defined at:
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
   Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

@keith, #3070 should fix your issue, I think. The toolchain pulling in query seems to be unrelated to pip integration, so if it works after #3070 and #3058 is merged, I think it is safe to say that the issue is resolved.

python/private/pypi/select_whl.bzl

aignas force-pushed the exp/pypi-simplify branch from 1d55c50 to c95dce6 Compare July 5, 2025 03:21

aignas changed the title ~~next: pypi stabilization of experimental_index_url~~ fix(pypi): pull fewer wheels with experimental_index_url Jul 5, 2025

aignas mentioned this pull request Jul 6, 2025

feat: freethreaded support for the builder API #3063

Draft

aignas force-pushed the exp/pypi-simplify branch from 415e893 to dc50a12 Compare July 6, 2025 12:52

aignas added 4 commits July 6, 2025 22:17

Merge branch 'test/independent-platforms' into exp/pypi-simplify

855a673

aignas force-pushed the exp/pypi-simplify branch from dc50a12 to 855a673 Compare July 6, 2025 13:42

aignas mentioned this pull request Jul 6, 2025

chore(pypi): remove unused config setting code #3065

Draft

rickeylev reviewed Jul 6, 2025

View reviewed changes

aignas added 8 commits July 7, 2025 18:30

Merge branch 'main' into exp/pypi-simplify

916556a

wip: add notes fix bugs and address comments

818f3e3

extra fixes and add a helper for python_tag

b00d7a2

more cleanup

bb838c0

clarify the doc

eb2de67

rename

233ca9d

remove merge conflicts

0991ad1

add a test for a bug

9553502

This was referenced Jul 7, 2025

experimental_index_url slows down downloads by 25-50% #2849

Open

"Illegal ambiguous match" error with experimental_index_url #2759

Open

aignas added 3 commits July 7, 2025 22:35

remove lower tier platforms from pip.default testing

a4566b1

fix an edge case

c22e3d5

fix docs building

b40f6a4

aignas marked this pull request as draft July 7, 2025 23:50

add back the macosx x86_64 because our CI depends on it

0bd5fbf

aignas marked this pull request as ready for review July 8, 2025 04:09

aignas mentioned this pull request Jul 8, 2025

Create a test that tests whl_library with an sdist #3071

Open

aignas added 9 commits July 8, 2025 18:25

Merge branch 'main' into exp/pypi-simplify

88c8959

add more changelog

95a8423

fix defaults and add docs

a0b7f43

revert an unnecessary diff

6d6276b

add back the failure

29c82d1

undo an unnecessary change

a8d67ed

move a note

3ec5225

cleanup

a19bb08

add notes about defaults

d36610f

hartikainen reviewed Jul 8, 2025

View reviewed changes

python/private/pypi/select_whl.bzl Show resolved Hide resolved

aignas added 2 commits July 12, 2025 23:17

correctly select by py_tag

6a48f85

Merge branch 'main' into exp/pypi-simplify

7ebf0c0

aignas requested a review from hartikainen July 12, 2025 14:19

aignas added 4 commits July 12, 2025 23:50

refactor/cleanup code

281f7fc

cleanup

a53c1e8

add a note

82e1532

Merge branch 'main' into exp/pypi-simplify

548bd54

aignas force-pushed the exp/pypi-simplify branch from 3d13783 to 548bd54 Compare July 15, 2025 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(pypi): pull fewer wheels with experimental_index_url #3058

fix(pypi): pull fewer wheels with experimental_index_url #3058

Uh oh!

aignas commented Jul 5, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 5, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickeylev Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 7, 2025 •

edited

Loading

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix(pypi): pull fewer wheels with experimental_index_url #3058

Are you sure you want to change the base?

fix(pypi): pull fewer wheels with experimental_index_url #3058

Uh oh!

Conversation

aignas commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickeylev Jul 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

aignas commented Jul 5, 2025 •

edited

Loading

aignas commented Jul 5, 2025 •

edited

Loading

keith commented Jul 7, 2025 •

edited

Loading

aignas commented Jul 7, 2025 •

edited

Loading