"Cross compilation" of py_binary/py_image/py_library targets #260

nimish · 2019-11-15T23:53:23Z

Hi,

I have a py_binary that depends on a python pip library (grpcio) that has a native extension bundled in. This means that to create a linux container i'd need to have the pip_import rule download the manylinux wheel, not the host one (macos in my case).

Is there a way to force this? Otherwise py_image will happily just bundle up wheels with darwin native libs. py_binary will also only make host-runnable things.

ali5h · 2019-12-16T07:57:16Z

use https://github.com/ali5h/rules_pip/ and use pip_install(["--platform=linux_x86_64"])

nimish · 2019-12-16T20:59:57Z

@ali5h this works (thanks!) but is there integration into the bazel platform selection functions? I don't want separate targets for Linux and Mac.

E.g. Something that works with https://docs.bazel.build/versions/master/platforms.html

ali5h · 2019-12-16T21:55:02Z

you can define multiple piplib repos for different platforms and use select to pickup correct one

nimish · 2019-12-16T22:06:24Z

Is that documented and supported anywhere? That would be the ideal case, for bazel to automatically pick up the right pip repo for the right target platform.

E: should it not just be built in to the py_binary/py_library rule, to select the right target platform libs automatically?

RemiTorracinta · 2020-04-05T19:38:50Z

Would really, really like to see this as well.

github-actions · 2021-04-14T22:52:30Z

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

brduarte · 2021-08-06T12:53:17Z

Help

UebelAndre · 2021-10-22T16:10:19Z

Seems #510 and #531 and addressing the same issue? I'd love to have that functionality so excited to so folks interested enough to open PRs 😄

Hopefully @alexeagle and @meastham can combine forces to get the awesome functionality over the line! 🚀

UebelAndre · 2021-12-08T10:21:25Z

Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃

UebelAndre · 2022-01-07T05:24:11Z

I wonder if pipenv could be used to facilitate generating cross platform dependency graphs. Does anyone have experience with the tool?

meastham · 2022-01-09T14:42:33Z

Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃

Sorry for the long delay on this! I did some investigation on some of the open questions, I'll see if we can find a path to getting this merged.

I don't think the PRs are exactly addressing the same issue. #531 allows having different requirements files for different platforms, but doesn't not allow downloading wheels for a different platform than the host platform. #510 allows downloading wheels for arbitrary platforms ("cross compiling"), but requires every platform to share one requirements file which has potential problems. Conceptually they could be compatible but the implementations would need to be modified to harmonize the way platform selection is done.

jvolkman · 2022-02-16T19:24:52Z

I'm all for the wheel-only approach taken in #510. I don't want native compilation happening at all in the loading phase, as it can lead to hard-to-diagnose cache misses between machines (in my experience).

Thinking out loud here. If adding a wheel-only option, can we remove pip from the build-time process entirely? Instead of using pip download, why not just use bazel's built-in http tools to download the wheel? Instead of using pip-compile to generate a locked requirements.txt, write something that generates a .bzl file with compiled dependencies in a more bazel-centric fashion. Maybe resolvers from poetry or pipenv are used for this, which apparently support multi-platform resolve. Dependencies between libraries - including platform specific - could be explicit in the generated file and defined using bazel's own select or whatever.

I'm sure there are gotchas here (and significant work), but maybe detaching from pip could open up new avenues.

alexeagle · 2022-02-16T23:50:36Z

I think it should also be possible to compile python/C++ sources into wheels, however those need to happen in actions so they are debuggable and so that the target platform can be used.

gopher-maker · 2022-04-15T00:11:20Z

Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!

meastham · 2022-04-20T14:46:05Z

Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!

Hey @gopher-maker,

I'm just blocked on getting some guidance from a repository owner on how to resolve the outstanding issues with #510 (not a complaint btw, I'm sure everybody is quite busy!).

FWIW we've been using it for a fairly large Python codebase without significant problems for about 9 months now, so if you're feeling adventurous you can use it already. It looks like it now needs some non-trivial rebasing work; I'll see if I can get to that this week.

Related issue: bazel-contrib/rules_python#260

adeeb10abbas · 2023-03-14T02:49:29Z

Any leads on this yet? I am running into a similar issue trying to use Mujoco in a bazel workspace.

alexeagle · 2023-03-15T14:21:20Z

@f0rmiga has been working on something related to this at a client, I don't have any update sorry.

f0rmiga · 2023-03-16T02:24:19Z

There's a current effort that @philsc is writing in a doc, and has the collaboration from @jvolkman. I wrote a resolver to download Python packages using http_file. It's similar to how Gazelle does it for Go third-party deps. I'll start a draft PR in the coming days to have it maintained in rules_python. It will change quite a bit the current workflow to work with wheels in Bazel, and I don't have all the answers yet, so I don't foresee it being in a release soon.

This PR starts using the newly introduced (#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards #260, #2319, #2241 Fixes #2423

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423 fix(toolchain) Override coverage rc

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423

This PR moves the parsing of `Requires-Dist` to the analysis phase within the `whl_library_targets_from_requires` macro. The original `whl_library_targets` macro has been left unchanged so that I don't have to reinvent the unit tests - it is well covered under tests. Before this PR we had to wire the `target_platforms` via the `experimental_target_platforms` attr in the `whl_library`, which means that whenever this would change (e.g. the minor Python version changes), the wheel would be re-extracted even though the final result may be the same. This also cleans up the code by removing left over TODO notes or code that no longer make sense. Work towards bazel-contrib#260, bazel-contrib#2319

This PR moves the parsing of `Requires-Dist` to the loading phase within the `whl_library_targets_from_requires` macro. The original `whl_library_targets` macro has been left unchanged so that I don't have to reinvent the unit tests - it is well covered under tests. Before this PR we had to wire the `target_platforms` via the `experimental_target_platforms` attr in the `whl_library`, which means that whenever this would change (e.g. the minor Python version changes), the wheel would be re-extracted even though the final result may be the same. This refactor uncovered that the dependency graph creation was incorrect if we had multiple target Python versions due to various heuristics that this had. In hindsight I had them to make the generated `BUILD.bazel` files more readable when the unit test coverage was not great. Now this is unnecessary and since everything is happening in Starlark I thought that having a simpler algorithm that does the right thing always is the best way. This also cleans up the code by removing left over TODO notes or code that no longer make sense. Work towards #260, #2319

This PR moves the parsing of `Requires-Dist` to the loading phase within the `whl_library_targets_from_requires` macro. The original `whl_library_targets` macro has been left unchanged so that I don't have to reinvent the unit tests - it is well covered under tests. Before this PR we had to wire the `target_platforms` via the `experimental_target_platforms` attr in the `whl_library`, which means that whenever this would change (e.g. the minor Python version changes), the wheel would be re-extracted even though the final result may be the same. This refactor uncovered that the dependency graph creation was incorrect if we had multiple target Python versions due to various heuristics that this had. In hindsight I had them to make the generated `BUILD.bazel` files more readable when the unit test coverage was not great. Now this is unnecessary and since everything is happening in Starlark I thought that having a simpler algorithm that does the right thing always is the best way. This also cleans up the code by removing left over TODO notes or code that no longer make sense. Work towards #260, #2319 (cherry picked from commit a19e1e4)

This is a flag to start leveraging of the new codepahs. Work towards #260

This is a flag to start leveraging of the new code paths. The Starlark implementation has been added in 1.4 and has been reverted in the latest release candidates. The `env` variable will be a good way to roll it out more gradually and get more testing. For now we are switching only the `whl_library` internals as the `requirements.txt` files from `uv` may use `*` in `python_full_version` and `platform_version` that are not yet fully supported (#2826). Main goals for this is to start using Starlark implementation so that we don't have any hidden variables. What is more, having this in Starlark is the most maintainable long-term solution for supporting cross-platform builds. Work towards #260 --------- Co-authored-by: Richard Levasseur <[email protected]>

surlyengineer mentioned this issue Nov 30, 2019

Invalid ELF header - py_image with binary dependencies on OS X bazelbuild/rules_docker#372

Closed

cwaeland mentioned this issue Aug 19, 2020

Configurable download of cross platform extensions ali5h/rules_pip#48

Closed

github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Apr 14, 2021

thundergolfer mentioned this issue May 9, 2021

pip_import/pip_install doesn't support cross-compilation #97

Closed

thundergolfer removed the Can Close? Will close in 30 days if there is no new activity label May 9, 2021

thundergolfer self-assigned this May 9, 2021

thundergolfer mentioned this issue May 9, 2021

Proper (cross) compilation of pip packages #453

Closed

meastham mentioned this issue Aug 14, 2021

Add support for multi-platform pip download to pip_parse #510

Closed

13 tasks

UebelAndre mentioned this issue Jan 6, 2022

Fix false positive tests in CI for Windows #597

Closed

12 tasks

kchodnicki mentioned this issue Jan 13, 2022

pip_install(["--platform linux_x86_64"] running pip wheel and result in no such option: --platform linux_x86_64 ali5h/rules_pip#68

Closed

laurynaslubys mentioned this issue Apr 29, 2022

RFC Better transitions bazelbuild/rules_docker#2073

Closed

stephane-caron added a commit to stephane-caron/vulp that referenced this issue May 27, 2022

Disable PyPI dependencies for the Raspberry Pi

7d82d40

Related issue: bazel-contrib/rules_python#260

aignas mentioned this issue Apr 6, 2025

feat: add support for passing extra pip args by platform #2745

Open

aignas mentioned this issue Apr 16, 2025

fix: load target_platforms through the hub #2781

Merged

aignas mentioned this issue May 2, 2025

test: compare Starlark and Python deps parsing for host_plat #2851

Open

aignas added a commit that referenced this issue May 3, 2025

feat: add an env variable to toggle pipstar

51140bc

This is a flag to start leveraging of the new codepahs. Work towards #260

aignas mentioned this issue May 3, 2025

feat: add an env variable to toggle pipstar #2855

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Cross compilation" of py_binary/py_image/py_library targets #260

"Cross compilation" of py_binary/py_image/py_library targets #260

nimish commented Nov 15, 2019

ali5h commented Dec 16, 2019

nimish commented Dec 16, 2019

ali5h commented Dec 16, 2019

nimish commented Dec 16, 2019 •

edited

Loading

RemiTorracinta commented Apr 5, 2020

github-actions bot commented Apr 14, 2021

brduarte commented Aug 6, 2021

UebelAndre commented Oct 22, 2021

UebelAndre commented Dec 8, 2021

UebelAndre commented Jan 7, 2022

meastham commented Jan 9, 2022 •

edited

Loading

jvolkman commented Feb 16, 2022

alexeagle commented Feb 16, 2022

gopher-maker commented Apr 15, 2022

meastham commented Apr 20, 2022

adeeb10abbas commented Mar 14, 2023

alexeagle commented Mar 15, 2023

f0rmiga commented Mar 16, 2023 •

edited

Loading

"Cross compilation" of py_binary/py_image/py_library targets #260

"Cross compilation" of py_binary/py_image/py_library targets #260

Comments

nimish commented Nov 15, 2019

ali5h commented Dec 16, 2019

nimish commented Dec 16, 2019

ali5h commented Dec 16, 2019

nimish commented Dec 16, 2019 • edited Loading

RemiTorracinta commented Apr 5, 2020

github-actions bot commented Apr 14, 2021

brduarte commented Aug 6, 2021

UebelAndre commented Oct 22, 2021

UebelAndre commented Dec 8, 2021

UebelAndre commented Jan 7, 2022

meastham commented Jan 9, 2022 • edited Loading

jvolkman commented Feb 16, 2022

alexeagle commented Feb 16, 2022

gopher-maker commented Apr 15, 2022

meastham commented Apr 20, 2022

adeeb10abbas commented Mar 14, 2023

alexeagle commented Mar 15, 2023

f0rmiga commented Mar 16, 2023 • edited Loading

nimish commented Dec 16, 2019 •

edited

Loading

meastham commented Jan 9, 2022 •

edited

Loading

f0rmiga commented Mar 16, 2023 •

edited

Loading