fix(pypi): pull fewer wheels with experimental_index_url #3058
+1,014
−607
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Created this as WIP to signal that I am working on this and to invite feedback. The CI should be mostly happy but there are a few TODOs before we can fully merge and I want to test it a little bit more.
TODO cleanup after this PR is proven to not break things:
|
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
|
The CI failure is a flake. |
This splits the configuration that we have for the `rules_python` and the defaults that we set for our users from the actual unit tests where we check that the extension is working correctly. With this we will be able to dog food the API and point users into the `MODULE.bazel` as the example snippet. Work towards bazel-contrib#2949 Work towards bazel-contrib#2747
Before it seems that we were relying on particular names in the pipstar platforms. This ensures that we rely on this less. Whilst at it fix a few typos and improve the formatting of the code. Work towards bazel-contrib#2949 Work towards bazel-contrib#2747
Before this we would pull all of the wheels that the user target configuration would be compatible with and that meant that it was not customizable. This also meant that there were a lot of footguns in the configuration where the select statements were not really foolproof. With this PR we select only those sources that need to be for the declared configurations. Freethreaded support should be done by defining extre freethreaded platforms using the new builder API. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
|
This would be ready to be reviewed, but because it is stacked on two PRs, let's get those out of the way first. |
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.
aignas
added a commit
to aignas/rules_python
that referenced
this pull request
Jul 6, 2025
Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.
rickeylev
reviewed
Jul 6, 2025
python/private/pypi/extension.bzl
Outdated
| )) | values.env | ||
| platforms[key] = struct( | ||
| env = env_, | ||
| want_abis = [ |
There was a problem hiding this comment.
What's the difference between the abi computed above, and the want_abis here?
hartikainen
reviewed
Jul 8, 2025
3d13783 to
548bd54
Compare
|
Just a quick note, this can be reviewed as is, but I want to change up the
Right now we just pick the last one in the list, which is a bug. If you have ideas how we can express this idea better, let me know. |
f50d6f9 to
0427182
Compare
|
@keith, @hartikainen, could you please retest this PR if it works with your setup. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this we would pull all of the wheels that the user target
configuration would be compatible with and that meant that it was not
customizable. This also meant that there were a lot of footguns in the
configuration where the select statements were not really foolproof.
With this PR we select only those sources that need to be for the
declared configurations.
Freethreaded support should be done by defining extra freethreaded
platforms using the new builder API. It is done as a followup in #3063.
This is also changing the default platforms to be only the fully supported
platforms. This makes the testing easier and avoids us running into
compatibility issues during the rollout.
Work towards #2747
Fixes #2759
Fixes #2849