refactor(pypi): implement PEP508 compliant marker evaluation #2692

aignas · 2025-03-23T09:30:34Z

This implements the PEP508 compliant marker evaluation in starlark and
removes the need for the Python interpreter when evaluating requirements
files passed to pip.parse. This makes the evaluation faster and allows
us to fix a few known issues (#2690).

In the future the intent is to move the METADATA parsing to pure
starlark so that the RequiresDist could be parsed in starlark at the
macro evaluation or analysis phases. This should make it possible to
more easily solve the design problem that more and more things need to
be passed to whl_library as args to have a robust dependency parsing:

[pypi] Support python micro version in env marker based dependency evaluation #2319 needs the full Python version to have correct cross-platform
compatible METADATA parsing and passing it to Python and back makes
it difficult/annoying to implement.
Parsing the METADATA file requires the precise list of target
platform or the list of available packages in the requirements.txt.
This means that without it we cannot trim the dependency tree in the
whl_library. Doing this at macro loading phase allows us to depend
on .bzl files in the hub_repository and more effectively pass
information.

I can remotely see that this could become useful in py_wheel or an building
wheels from sdists as the environment markers may be present in various source
metadata as well. What is more uv.lock file has the env markers as part of
the lock file information, so this might be useful there.

Work towards #2423
Work towards #260
Split from #2629

This implements the PEP508 compliant marker evaluation in starlark and removes the need for the Python interpreter when evaluating requirements files passed to `pip.parse`. This makes the evaluation faster and allows us to fix a few known issues (bazel-contrib#2690). In the future the intent is to move the `METADATA` parsing to pure starlark so that the `RequiresDist` could be parsed in starlark at the macro evaluation or analysis phases. This should make it possible to more easily solve the design problem that more and more things need to be passed to `whl_library` as args to have a robust dependency parsing: * bazel-contrib#2319 needs the full Python version to have correct cross-platform compatible METADATA parsing and passing it to `Python` and back makes it difficult/annoying to implement. * Parsing the `METADATA` file requires the precise list of target platform or the list of available packages in the `requirements.txt`. This means that without it we cannot trim the dependency tree in the `whl_library`. Doing this at macro loading phase allows us to depend on `.bzl` files in the `hub_repository` and more effectively pass information. Fixes bazel-contrib#2423

python/private/pypi/pep508_env.bzl

aignas · 2025-03-30T12:00:54Z

Refreshed the lockfile before and after the change:

bazel mod deps --lockfile_mode=refresh

And saw no difference in entries except for the expected:

$ diff MODULE.bazel.lock{before,after}
161c161
<         "bzlTransitiveDigest": "HtK14jTL6GRmVkjtcDvWJ+J5LmpFcDz5iGWjGt4RORc=",
---
>         "bzlTransitiveDigest": "1YngqOV37S3bLvhBu3wQ+h/bYrkyRpmh11ur6JIqqu0=",
164d163
<           "@@+internal_deps+pypi__packaging//packaging-24.0.dist-info/RECORD": "be1aea790359b4c2c9ea83d153c1a57c407742a35b95ee36d00723509f5ed5dd",
167,168d165
<           "@@//python/private/pypi/requirements_parser/resolve_target_platforms.py": "42bf51980528302373529bcdfddb8014e485182d6bc9d2f7d3bbe1f11d8d923d",
<           "@@//python/private/pypi/whl_installer/platform.py": "763d705ceaa41c87d8a6cfe8c6bc261a20881628b96f2fb4c4fef4b9befa53fe",

aignas · 2025-03-30T14:06:26Z

OK, I think this is ready to be merged. This is probably already higher fidelity/more robust than the Python impl of platform.py in our repo.

This PR starts using the newly introduced (#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards #260, #2319, #2241 Fixes #2423

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423 fix(toolchain) Override coverage rc

…ontrib#2629) This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant requirement marker parser in starlark and moves the dependency generation from the Python language (`whl_installer`) to the Starlark in the `whl_library` repository rule. This PR is (almost) a pure refactor where no bugs are fixed, but this is foundational work that also adds notes on how things will be moved to macros (i.e. analysis phase) so that we can fix a few long standing bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260). Refactor: * I have migrated all of the unit tests from Python to starlark for deps generation from METADATA `Requires-Dist` fields. * Read the `METADATA` file itself in Starlark. Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241 Fixes bazel-contrib#2423

Summary: - Revert to using Python for marker evaluation during parsing of requirements (partial revert of bazel-contrib#2692). - Use Python to parse whl METADATA. Fixes bazel-contrib#2830

Summary: - Revert to using Python for marker evaluation during parsing of requirements (partial revert of #2692). - Use Python to parse whl METADATA. - Bugfix the new simpler algorithm and add a new unit test. Fixes #2830

Summary: - Revert to using Python for marker evaluation during parsing of requirements (partial revert of #2692). - Use Python to parse whl METADATA. - Bugfix the new simpler algorithm and add a new unit test. Fixes #2830 (cherry picked from commit 5b9d545)

aignas requested review from rickeylev and groodt as code owners March 23, 2025 09:30

aignas added 3 commits March 23, 2025 18:35

fix tests

435b9d2

fix more tests

f7edb9e

Merge branch 'main' into feat/extension-eval-without-python

2f34976

aignas mentioned this pull request Mar 29, 2025

2025 Priorities #2520

Open

rickeylev approved these changes Mar 29, 2025

View reviewed changes

aignas commented Mar 29, 2025

View reviewed changes

python/private/pypi/pep508_env.bzl Outdated Show resolved Hide resolved

aignas added 4 commits March 30, 2025 08:07

Fix ppc handling

d4b19b4

Merge branch 'main' into feat/extension-eval-without-python

9f63485

wip

0585c0d

normalize the platform_machine during evaluation

a17aa55

aignas added 2 commits March 30, 2025 22:53

wip

aa4e055

refactor

f2ec55e

aignas enabled auto-merge March 30, 2025 14:06

aignas added this pull request to the merge queue Mar 30, 2025

Merged via the queue into bazel-contrib:main with commit bfad507 Mar 30, 2025
3 checks passed

aignas mentioned this pull request Apr 5, 2025

refactor(pypi): translate wheel METADATA parsing to starlark #2629

Merged

aignas mentioned this pull request Apr 28, 2025

revert(pypi): use Python for marker eval and METADATA parsing #2834

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(pypi): implement PEP508 compliant marker evaluation #2692

refactor(pypi): implement PEP508 compliant marker evaluation #2692

aignas commented Mar 23, 2025 •

edited

Loading

aignas commented Mar 30, 2025

aignas commented Mar 30, 2025

refactor(pypi): implement PEP508 compliant marker evaluation #2692

refactor(pypi): implement PEP508 compliant marker evaluation #2692

Conversation

aignas commented Mar 23, 2025 • edited Loading

aignas commented Mar 30, 2025

aignas commented Mar 30, 2025

aignas commented Mar 23, 2025 •

edited

Loading