Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Third party dependencies are incorrect when using RBE because host != exec #2241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
groodt opened this issue Sep 21, 2024 · 0 comments
Open

Third party dependencies are incorrect when using RBE because host != exec #2241

groodt opened this issue Sep 21, 2024 · 0 comments

Comments

@groodt
Copy link
Collaborator

groodt commented Sep 21, 2024 •
edited
Loading

Context

This is a tracking issue to recognise the lack of support for Remote Execution (RBE) when using the support for third party dependencies from PyPI (or similar indexes/mirrors).

A reproduction of the issue has been developed in this rules_python_rbe_exec_exploration repository.

The issue occurs when there are native dependencies in the transitive closure of third party dependencies, which is very common in the Python ecosystem.

The root cause of the issue is that the current implementation that imports third party dependencies as targets of the bazel build assumes that the host platform is the same as the execution platform (host == exec). However, a very common configuration of remote execution would be macos or windows host platforms executing remote builds on linux platforms (host != exec).

Known workarounds

  1. Only use pure python third party dependencies (this is not really practical)
  2. Try the experimental support for multi-platform wheel repositories using experimental_index_url (follow "Cross compilation" of py_binary/py_image/py_library targets #260 for when this becomes default)
  3. Use alternative rules such as rules_pycross that can import third party dependencies where host != exec

Related
@aignas aignas mentioned this issue Dec 24, 2024
Open
@aignas aignas pinned this issue Mar 21, 2025
@aignas aignas mentioned this issue Mar 22, 2025
Merged
@aignas aignas mentioned this issue Apr 5, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Apr 13, 2025
@aignas
refactor(pypi): translate wheel METADATA parsing to starlark (#2629)
2cb920c 
This PR starts using the newly introduced (#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards #260, #2319, #2241
Fixes #2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
bb56a77 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
6ca7e29 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
da95314 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
d10bf4c 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423

fix(toolchain) Override coverage rc
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
01d25ad 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423

fix(toolchain) Override coverage rc
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
3b61a5d 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
ffca3e4 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
05ef420 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this issue Apr 14, 2025
@aignas @ewianda
refactor(pypi): translate wheel METADATA parsing to starlark (bazel-c…
c277348 
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@groodt