Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 7de43d1

Browse files
authored
feat(bzlmod): support cross-platform whl setups within the main hub repo (#1837)
With this change we add support for platform-specific wheel registration and doing the selection of which wheel is used at build time. This supports: - Different package versions for different platforms. - Use string_flags to configure what to fetch/select: - only whls, only sdist or auto mode. - libc version and `musl` vs `glibc` selection. - universal2 vs arch wheels for mac. - target osx version selection. Summary of changes: - The `uv pip install` would only warn the user of yanked packages but would not refuse to install them. Update our implementation to better match the same behaviour. - A previous PR has added the support for passing it in the `requirements_by_platform` and this just add the necessary code to make sure that we can also do the dependency management when parsing the `whl` `METADATA` files. - Only configure `dev_pip` deps for `linux` and `osx` platforms to not raise issues later. - Add a function for generating a `whl_library` name from a `filename`. - Add a macro for generating all config settings for a particular set of parameters. - Update `render_pkg_aliases` to also use those config settings. - Update the toolchain selection `target_settings` to use the `py_linux_libc` config setting. With this the user can register a `musl` linux toolchain if needed. We can also use similar `flag_values` to resolve #1876. - Integrate everything in the `pip` extension and setup cross-platform select statements. Work towards #1357, #260 Stacked on #1937
1 parent e42f8f4 commit 7de43d1

32 files changed

+2682
-378
lines changed

CHANGELOG.md

+26-1
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,20 @@ A brief description of the categories of changes:
3636
* (bzlmod): The `python` and internal `rules_python` extensions have been
3737
marked as `reproducible` and will not include any lock file entries from now
3838
on.
39-
4039
* (gazelle): Remove gazelle plugin's python deps and make it hermetic.
4140
Introduced a new Go-based helper leveraging tree-sitter for syntax analysis.
4241
Implemented the use of `pypi/stdlib-list` for standard library module verification.
42+
* (pip.parse): Do not ignore yanked packages when using `experimental_index_url`.
43+
This is to mimic what `uv` is doing. We will print a warning instead.
44+
* (pip.parse): Add references to all supported wheels when using `experimental_index_url`
45+
to allowing to correctly fetch the wheels for the right platform. See the
46+
updated docs on how to use the feature. This is work towards addressing
47+
[#735](https://github.com/bazelbuild/rules_python/issues/735) and
48+
[#260](https://github.com/bazelbuild/rules_python/issues/260). The spoke
49+
repository names when using this flag will have a structure of
50+
`{pip_hub_prefix}_{wheel_name}_{py_tag}_{abi_tag}_{platform_tag}_{sha256}`,
51+
which is an implementation detail which should not be relied on and is there
52+
purely for better debugging experience.
4353

4454
### Fixed
4555
* (gazelle) Remove `visibility` from `NonEmptyAttr`.
@@ -63,6 +73,13 @@ A brief description of the categories of changes:
6373
the `experimental_index_url` feature which will fetch metadata from PyPI or a
6474
different private index and write the contents to the lock file. Fixes
6575
[#1643](https://github.com/bazelbuild/rules_python/issues/1643).
76+
* (pip.parse): Install `yanked` packages and print a warning instead of
77+
ignoring them. This better matches the behaviour of `uv pip install`.
78+
* (toolchains): Now matching of the default hermetic toolchain is more robust
79+
and explicit and should fix rare edge-cases where the host toolchain
80+
autodetection would match a different toolchain than expected. This may yield
81+
to toolchain selection failures when the python toolchain is not registered,
82+
but is requested via `//python/config_settings:python_version` flag setting.
6683

6784
### Added
6885
* (rules) Precompiling Python source at build time is available. but is
@@ -104,6 +121,14 @@ A brief description of the categories of changes:
104121
{obj}`PyRuntimeInfo.zip_main_template`.
105122
* (toolchains) A replacement for the Bazel-builtn autodetecting toolchain is
106123
available. The `//python:autodetecting_toolchain` alias now uses it.
124+
* (pip): Support fetching and using the wheels for other platforms. This
125+
supports customizing whether the linux wheels are pulled for `musl` or
126+
`glibc`, whether `universal2` or arch-specific MacOS wheels are preferred and
127+
it also allows to select a particular `libc` version. All of this is done via
128+
the `string_flags` in `@rules_python//python/config_settings`. If there are
129+
no wheels that are supported for the target platform, `rules_python` will
130+
fallback onto building the `sdist` from source. This behaviour can be
131+
disabled if desired using one of the available string flags as well.
107132

108133
[precompile-docs]: /precompiling
109134

MODULE.bazel

+3-1
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,9 @@ dev_pip.parse(
9696
},
9797
hub_name = "dev_pip",
9898
python_version = "3.11",
99-
requirements_lock = "//docs/sphinx:requirements.txt",
99+
requirements_by_platform = {
100+
"//docs/sphinx:requirements.txt": "linux_*,osx_*",
101+
},
100102
)
101103
dev_pip.parse(
102104
hub_name = "pypiserver",

docs/sphinx/api/python/config_settings/index.md

+82-8
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,18 @@
55

66
# //python/config_settings
77

8-
:::{bzl:flag} precompile
8+
:::{bzl:flag} python_version
9+
Determines the default hermetic Python toolchain version. This can be set to
10+
one of the values that `rules_python` maintains.
11+
:::
12+
13+
::::{bzl:flag} precompile
914
Determines if Python source files should be compiled at build time.
1015

11-
NOTE: The flag value is overridden by the target level `precompile` attribute,
16+
:::{note}
17+
The flag value is overridden by the target level `precompile` attribute,
1218
except for the case of `force_enabled` and `forced_disabled`.
19+
:::
1320

1421
Values:
1522

@@ -27,22 +34,24 @@ Values:
2734
* `force_disabled`: Like `disabled`, except overrides target-level setting. This
2835
is useful useful for development, testing enabling precompilation more
2936
broadly, or as an escape hatch if build-time compiling is not available.
30-
:::
37+
::::
3138

32-
:::{bzl:flag} precompile_source_retention
39+
::::{bzl:flag} precompile_source_retention
3340
Determines, when a source file is compiled, if the source file is kept
3441
in the resulting output or not.
3542

36-
NOTE: This flag is overridden by the target level `precompile_source_retention`
43+
:::{note}
44+
This flag is overridden by the target level `precompile_source_retention`
3745
attribute.
46+
:::
3847

3948
Values:
4049

4150
* `keep_source`: Include the original Python source.
4251
* `omit_source`: Don't include the orignal py source.
4352
* `omit_if_generated_source`: Keep the original source if it's a regular source
4453
file, but omit it if it's a generated file.
45-
:::
54+
::::
4655

4756
:::{bzl:flag} precompile_add_to_runfiles
4857
Determines if a target adds its compiled files to its runfiles.
@@ -59,15 +68,80 @@ Values:
5968
incrementally enabling precompilation on a per-binary basis.
6069
:::
6170

62-
:::{bzl:flag} pyc_collection
71+
::::{bzl:flag} pyc_collection
6372
Determine if `py_binary` collects transitive pyc files.
6473

65-
NOTE: This flag is overridden by the target level `pyc_collection` attribute.
74+
:::{note}
75+
This flag is overridden by the target level `pyc_collection` attribute.
76+
:::
6677

6778
Values:
6879
* `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
6980
* `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
81+
::::
82+
83+
::::{bzl:flag} py_linux_libc
84+
Set what libc is used for the target platform. This will affect which whl binaries will be pulled and what toolchain will be auto-detected. Currently `rules_python` only supplies toolchains compatible with `glibc`.
85+
86+
Values:
87+
* `glibc`: Use `glibc`, default.
88+
* `muslc`: Use `muslc`.
89+
:::{versionadded} 0.33.0
90+
:::
91+
::::
92+
93+
::::{bzl:flag} pip_whl
94+
Set what distributions are used in the `pip` integration.
95+
96+
Values:
97+
* `auto`: Prefer `whl` distributions if they are compatible with a target
98+
platform, but fallback to `sdist`. This is the default.
99+
* `only`: Only use `whl` distributions and error out if it is not available.
100+
* `no`: Only use `sdist` distributions. The wheels will be built non-hermetically in the `whl_library` repository rule.
101+
:::{versionadded} 0.33.0
70102
:::
103+
::::
104+
105+
::::{bzl:flag} pip_whl_osx_arch
106+
Set what wheel types we should prefer when building on the OSX platform.
107+
108+
Values:
109+
* `arch`: Prefer architecture specific wheels.
110+
* `universal`: Prefer universal wheels that usually are bigger and contain binaries for both, Intel and ARM architectures in the same wheel.
111+
:::{versionadded} 0.33.0
112+
:::
113+
::::
114+
115+
::::{bzl:flag} pip_whl_glibc_version
116+
Set the minimum `glibc` version that the `py_binary` using `whl` distributions from a PyPI index should support.
117+
118+
Values:
119+
* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
120+
* `X.Y`: The string representation of a `glibc` version. The allowed values depend on the `requirements.txt` lock file contents.
121+
:::{versionadded} 0.33.0
122+
:::
123+
::::
124+
125+
::::{bzl:flag} pip_whl_muslc_version
126+
Set the minimum `muslc` version that the `py_binary` using `whl` distributions from a PyPI index should support.
127+
128+
Values:
129+
* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
130+
* `X.Y`: The string representation of a `muslc` version. The allowed values depend on the `requirements.txt` lock file contents.
131+
:::{versionadded} 0.33.0
132+
:::
133+
::::
134+
135+
::::{bzl:flag} pip_whl_osx_version
136+
Set the minimum `osx` version that the `py_binary` using `whl` distributions from a PyPI index should support.
137+
138+
Values:
139+
* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
140+
* `X.Y`: The string representation of the MacOS version. The allowed values depend on the `requirements.txt` lock file contents.
141+
142+
:::{versionadded} 0.33.0
143+
:::
144+
::::
71145

72146
::::{bzl:flag} bootstrap_impl
73147
Determine how programs implement their startup process.

docs/sphinx/pip.md

+1-165
Original file line numberDiff line numberDiff line change
@@ -1,168 +1,4 @@
11
(pip-integration)=
22
# Pip Integration
33

4-
To pull in dependencies from PyPI, the `pip_parse` function is used, which
5-
invokes `pip` to download and install dependencies from PyPI.
6-
7-
In your WORKSPACE file:
8-
9-
```starlark
10-
load("@rules_python//python:pip.bzl", "pip_parse")
11-
12-
pip_parse(
13-
name = "pip_deps",
14-
requirements_lock = ":requirements.txt",
15-
)
16-
17-
load("@pip_deps//:requirements.bzl", "install_deps")
18-
19-
install_deps()
20-
```
21-
22-
For `bzlmod` an equivalent `MODULE.bazel` would look like:
23-
```starlark
24-
pip = use_extension("//python/extensions:pip.bzl", "pip")
25-
pip.parse(
26-
hub_name = "pip_deps",
27-
requirements_lock = ":requirements.txt",
28-
)
29-
use_repo(pip, "pip_deps")
30-
```
31-
32-
You can then reference installed dependencies from a `BUILD` file with:
33-
34-
```starlark
35-
load("@pip_deps//:requirements.bzl", "requirement")
36-
37-
py_library(
38-
name = "bar",
39-
...
40-
deps = [
41-
"//my/other:dep",
42-
"@pip_deps//requests",
43-
"@pip_deps//numpy",
44-
],
45-
)
46-
```
47-
48-
The rules also provide a convenience macro for translating the entries in the
49-
`requirements.txt` file (e.g. `opencv-python`) to the right bazel label (e.g.
50-
`@pip_deps//opencv_python`). The convention of bazel labels is lowercase
51-
`snake_case`, but you can use the helper to avoid depending on this convention
52-
as follows:
53-
54-
```starlark
55-
load("@pip_deps//:requirements.bzl", "requirement")
56-
57-
py_library(
58-
name = "bar",
59-
...
60-
deps = [
61-
"//my/other:dep",
62-
requirement("requests"),
63-
requirement("numpy"),
64-
],
65-
)
66-
```
67-
68-
If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation.
69-
70-
[whl_ep]: https://packaging.python.org/specifications/entry-points/
71-
72-
(per-os-arch-requirements)=
73-
## Requirements for a specific OS/Architecture
74-
75-
In some cases you may need to use different requirements files for different OS, Arch combinations. This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the `pip_parse` repository rule. The keys of the dictionary are labels to the file and the values are a list of comma separated target (os, arch) tuples.
76-
77-
For example:
78-
```starlark
79-
# ...
80-
requirements_by_platform = {
81-
"requirements_linux_x86_64.txt": "linux_x86_64",
82-
"requirements_osx.txt": "osx_*",
83-
"requirements_linux_exotic.txt": "linux_exotic",
84-
"requirements_some_platforms.txt": "linux_aarch64,windows_*",
85-
},
86-
# For the list of standard platforms that the rules_python has toolchains for, default to
87-
# the following requirements file.
88-
requirements_lock = "requirements_lock.txt",
89-
```
90-
91-
In case of duplicate platforms, `rules_python` will raise an error as there has
92-
to be unambiguous mapping of the requirement files to the (os, arch) tuples.
93-
94-
An alternative way is to use per-OS requirement attributes.
95-
```starlark
96-
# ...
97-
requirements_windows = "requirements_windows.txt",
98-
requirements_darwin = "requirements_darwin.txt",
99-
# For the remaining platforms (which is basically only linux OS), use this file.
100-
requirements_lock = "requirements_lock.txt",
101-
)
102-
```
103-
104-
(vendoring-requirements)=
105-
## Vendoring the requirements.bzl file
106-
107-
In some cases you may not want to generate the requirements.bzl file as a repository rule
108-
while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module
109-
such as a ruleset, you may want to include the requirements.bzl file rather than make your users
110-
install the WORKSPACE setup to generate it.
111-
See https://github.com/bazelbuild/rules_python/issues/608
112-
113-
This is the same workflow as Gazelle, which creates `go_repository` rules with
114-
[`update-repos`](https://github.com/bazelbuild/bazel-gazelle#update-repos)
115-
116-
To do this, use the "write to source file" pattern documented in
117-
https://blog.aspect.dev/bazel-can-write-to-the-source-folder
118-
to put a copy of the generated requirements.bzl into your project.
119-
Then load the requirements.bzl file directly rather than from the generated repository.
120-
See the example in rules_python/examples/pip_parse_vendored.
121-
122-
123-
(credential-helper)=
124-
## Credential Helper
125-
126-
The "use Bazel downloader for python wheels" experimental feature includes support for the Bazel
127-
[Credential Helper][cred-helper-design].
128-
129-
Your python artifact registry may provide a credential helper for you. Refer to your index's docs
130-
to see if one is provided.
131-
132-
See the [Credential Helper Spec][cred-helper-spec] for details.
133-
134-
[cred-helper-design]: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md
135-
[cred-helper-spec]: https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md
136-
137-
138-
### Basic Example:
139-
140-
The simplest form of a credential helper is a bash script that accepts an arg and spits out JSON to
141-
stdout. For a service like Google Artifact Registry that uses ['Basic' HTTP Auth][rfc7617] and does
142-
not provide a credential helper that conforms to the [spec][cred-helper-spec], the script might
143-
look like:
144-
145-
```bash
146-
#!/bin/bash
147-
# cred_helper.sh
148-
ARG=$1 # but we don't do anything with it as it's always "get"
149-
150-
# formatting is optional
151-
echo '{'
152-
echo ' "headers": {'
153-
echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]'
154-
echo ' }'
155-
echo '}'
156-
```
157-
158-
Configure Bazel to use this credential helper for your python index `example.com`:
159-
160-
```
161-
# .bazelrc
162-
build --credential_helper=example.com=/full/path/to/cred_helper.sh
163-
```
164-
165-
Bazel will call this file like `cred_helper.sh get` and use the returned JSON to inject headers
166-
into whatever HTTP(S) request it performs against `example.com`.
167-
168-
[rfc7617]: https://datatracker.ietf.org/doc/html/rfc7617
4+
See [PyPI dependencies](./pypi-dependencies).

0 commit comments

Comments
 (0)