Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support universal requirements for multiple python versions #2797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
keith opened this issue Apr 21, 2025 · 4 comments
Open

Support universal requirements for multiple python versions #2797

keith opened this issue Apr 21, 2025 · 4 comments
Labels
type: pip pip/pypi integration

Comments

@keith
Copy link
Member

keith commented Apr 21, 2025 •
edited
Loading

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

I don't think so

Description

Some dependencies can vary based on the current python version. In this case for requirements.txt files generated by uv or similar tools, the python version is encoded in the requirements.txt:

numpy==2.0.2 ; python_full_version < '3.10'
    # via -r requirements.in
numpy==2.2.5 ; python_full_version >= '3.10'
    # via -r requirements.in

In this case if you attempt to use this requirements file with pip.parse you get this error:

ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl:224:25: Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 607, column 25, in _pip_impl
                mods = parse_modules(module_ctx)
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 478, column 36, in parse_modules
                out = _create_whl_repos(
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 224, column 25, in _create_whl_repos
                fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_311_numpy_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for numpy

Ideally rules_python would know that there should only be a single repo here since these constraints are mutually exclusive

🔬 Minimal Reproduction

bazel build ... in #2798

🌍 Your Environment

Operating System:

  
linux x86_64

Output of bazel version:

  
8.2.1

Rules_python version:

c981569
keith added a commit to keith/rules_python that referenced this issue Apr 21, 2025
@keith
Repro for bazel-contrib#2797
bbb7ac9
@keith keith mentioned this issue Apr 21, 2025
Closed
@aignas
Copy link
Collaborator

aignas commented Apr 22, 2025

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

@aignas aignas added the type: pip pip/pypi integration label Apr 22, 2025
@baxelrod-bdai
Copy link

I am also getting this error.

Minimal reproduction

requirements.in has just:

ipython >= 5

and BUILD.bazel has:

load("@rules_uv//uv:pip.bzl", "pip_compile")

pip_compile(
    name = "generate_requirements",
    extra_args = [
        "--universal",
    ],
    requirements_in = "requirements.in",
    requirements_txt = "requirements.lock",
)

bazel run //:generate_requirements runs fine, and creates a lock file with:

ipython==8.35.0 ; python_full_version < '3.11' \
    --hash=sha256:d200b7d93c3f5883fc36ab9ce28a18249c7706e51347681f80a0aef9895f2520 \
    --hash=sha256:e6b7470468ba6f1f0a7b116bb688a3ece2f13e2f94138e508201fad677a788ba
    # via -r requirements.in
ipython==9.1.0 ; python_full_version >= '3.11' \
    --hash=sha256:2df07257ec2f84a6b346b8d83100bcf8fa501c6e01ab75cd3799b0bb253b3d2a \
    --hash=sha256:a47e13a5e05e02f3b8e1e7a0f9db372199fe8c3763532fe7a1e0379e4e135f16
    # via -r requirements.in

But when you try to bazel run anything else, I get the crash:

ERROR: /home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl:231:25: Traceback (most recent call last):
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 611, column 25, in _pip_impl
		mods = parse_modules(module_ctx)
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 482, column 36, in parse_modules
		out = _create_whl_repos(
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 231, column 25, in _create_whl_repos
		fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_310_ipython_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for ipython
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip

Versions

Bazelisk version: v1.19.0
Build label: 8.1.1

bazel_dep(name = "rules_python", version = "1.4.0-rc1")
bazel_dep(name = "rules_uv", version = "0.68.0")

Linux baxelrod-ThinkPad 6.8.0-57-generic #59~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Mar 19 17:07:41 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Notes

I tried adding this to my .bazelrc to force things to use a single Python version, but no luck.

build --action_env=PYTHON_VERSION=3.10
run --action_env=PYTHON_VERSION=3.10

The problem goes away if I don't specify --universal in the uv pip compile arguments. Because then no markers are added in the lock file.

@baxelrod-bdai
Copy link

@aignas can you explain your comment? Is this a workaround?

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

@keith
Copy link
Member Author

keith commented Apr 24, 2025

with your requirements file, since it has hashes, if you set experimental_index_url = "https://pypi.org/simple", on pip.parse it will not fail with this error.

In my case it seems using experimental_index_url causes a 50% regression in dependency download time, but I plan to file a separate issue for that.

I am also curious if the plan is to require this codepath for this fix, in which case maybe we should add a more clear error

@baxelrod-bdai baxelrod-bdai mentioned this issue Apr 24, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: pip pip/pypi integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aignas @keith @baxelrod-bdai