feat: data and pyi files in the venv #2936

aignas · 2025-05-27T14:19:11Z

This adds the remaining of the files into the venv and should get us
reasonably close to handling 99% of the cases.

The expected differences from this and a venv built by uv would be:

The RECORD files are excluded from the venvs for better cache hit
rate in bazel.

Topological ordering is removed because topo ordering doesn't provide
the "closer target first" guarantees desired. For now, just use default
ordering and document conflicts as undefined behavior. Internally, it
continues to use first-wins (i.e. first in depset.to_list() order) semantics.

Work towards #2156

This adds the necessary `.dist-info` files into the mix and should get us reasonably close to handling 99% of the cases. The expected differences from this and a `venv` built by `uv` would be: * Shared libraries are stored in `<package>.libs` in `uv` venvs. This can be achieved in `rules_python` by changing the `installer` settings in the `wheel_installer/wheel.py#unzip` function. * The `RECORD` files are excluded from the `venv`s for better cache hit rate in `bazel`, however I am not sure if we should do that for actual wheels that are downloaded from the internet. Tested: - [x] Building the `//docs` and manually checking the symlinks. - [ ] Unit tests Work towards bazel-contrib#2156

aignas · 2025-05-28T14:17:52Z

I am thinking I should also link other things passed as data for completeness.

I think excluding anything outside site-packages directory for now would be wise.

Tested:

Building the //docs and manually checking the symlinks.
Ensure that the .pyi files get included.
Ensure that the libs get included.
Tests that .dist_info gets included.
Tests that ensure that there are files only from one version of the package.
Remove the topological order usage and document the undefined behaviour.
document PyInfo changes.

aignas · 2025-05-30T01:14:01Z

I'll push some of the changes later but I have a hard time writing a test that would fullfill some of the requirements in the PR todo list.

python/private/py_executable.bzl

rickeylev

re: "src" field name: Let's pick a better name. "src" is an established name in bazel/starlark, so I don't think it should be overloaded.

package
dist
link_group
group_name
group
group_key1 (e.g. "simple"), group_key2 (e.g. "1.0.0")
dist, version (simple, 1.0)

?

The basic change in logic here is associating a set of paths with a dist-info name, right? And only one of those sets of paths will be used? The idea being, given:

("foo", <foo v1 paths>)
("foo", <foo v2 paths>)

Only one of those tuples of info is used.

tests/modules/other/simple_v2/site-packages/simple.libs/data.txt

tests/venv_site_packages_libs/bin.py

python/private/py_executable.bzl

…xternal data inclusion.

rickeylev · 2025-06-02T16:01:19Z

python/private/py_executable.bzl

@@ -682,15 +682,30 @@ def _create_venv_symlinks(ctx, venv_dir_map):
 def _build_link_map(entries):
    # dict[str kind, dict[str rel_path, str link_to_path]]
    link_map = {}
+
+    # Here we store venv paths by package


The type description makes it clear they're keyed by package, so no need to restate it in prose.

Suggested change

# Here we store venv paths by package

python/private/py_executable.bzl

python/private/py_library.bzl

rickeylev · 2025-06-02T16:22:28Z

python/private/py_library.bzl

+            venv_path = dist_info_dir,
+        ))
+
+    for src in ctx.files.srcs + ctx.files.data:


Something seems wrong here, but I can't quite put my finger on it.

Files in srcs have to have the init.py logic applied to detect namespace package boundaries to auto-detect the proper paths to create.

Files in data shouldn't be part of that logic. If the file is within a directory covered by something in srcs (i.e *.py), then there's no need to look at the data file. If the data file isn't in a directory covered by something in srcs, then it shouldn't treat a .so/.pyi/.pyc file as some sort of boundary marker.

Talking about special treatment, I don't think I understand why we need it. We could just treat everything as just files and interleave the directories? There is nothing special in my mind here.

Beforehand we had the auto-generated namespace pkg __init__.py files, but if we get rid of them everything is just files, right?

treat everything as files and interleave the directories

You mean, for every file, create a venv_symlink entry? That won't scale because it doubles the number of files a binary has to materialize, when all that's really needed is e.g. one symlink to a directory.

However, because of namespace packages, we can't simply use the directories directly under site-packages/.

What if we treat everything as namespace packages? The way I understand this loop is:

For each file in the sources, symlink their directories.

If we have common things across data and srcs then we double the number of directories that we need to process, but I don't see a way to work around the case where we have a big package (e.g. airflow) that have data files under site-packages/airflow/data-files-without-python-files that would need to be symlinked in addition to other namespace packages.

I am wondering how we can do this?

python/private/py_info.bzl

aignas · 2025-06-05T23:49:19Z

In my test I get the following venv_paths when the topological order deps are converted to list (if I remove the patch I have to sort the deps before merging the depsets):

simple
simple-1.0.0.dist-info
simple_v1_extras
simple
simple-2.0.0.dist-info
simple.libs
nspkg/subnspkg/alpha
nspkg/subnspkg/beta
nspkg/subnspkg/delta
nspkg/subnspkg/gamma
__init__.py
single_file.py
with_external_data.py
external_data

This means that It is depth first topological rather than breadth first, if that makes sense.

aignas · 2025-06-06T14:17:56Z

tests/venv_site_packages_libs/depset_tests.bzl

+            "b",
+            "c1",
+            "c2",
+        ],


OK, so I had an idea that instead of discussing in markdown about theory and findings from print debugs, we can have a Starlark test checking the depset behaviour.

So as I say, it seems that the topological sorting is DFS.

This reverts commit 507985b.

aignas · 2025-06-07T14:50:43Z

OK, PTAL, have updated docs.

rickeylev · 2025-06-07T19:15:35Z

I revisited what topological ordering means. You're right, essentially. It's not breadth-depth like I originally thought. It's about ensuring that the nodes that point to a node are listed before that node in the linearized ordering. Note that "nodes" means the values in the depset, not depset objects themselves. i.e. depset(["a"], transitive=["b", "a"]) results in [b, a].

Stated another way: there's no guarantee a value will occur before (or after) another when there is no path between them. This means, in order to get the semantics we want (via depset ordering) -- that simple_v2 can be overridden by another target -- the overriding target must depend on simple_v2, which kinda defeats the purpose of overriding (I shouldn't have to depend on what I want to remove in order to remove it).

So yeah, topological ordering doesn't give the "closet target wins" behavior that I was originally thinking.

Reading https://bazel.build/rules/lib/builtins/depset, preorder might be closer to what we want (direct elements first)? Though, topological ordering could maybe reflect an install order? I'll take the thinking-out-loud to slack.

I'm -1 on trying to classify something as 3P vs 1P. There just isn't a clear definition of what is or isn't. Whether code is made available via pip_parse, or I write an equivalent target in my main repo, or I get it from a custom repository rule, or etc doesn't tell if a particular target is "mine" or "not mine".

I would like to make progress on this PR, though. I'm thinking we just drop the mention of an explicit ordering to allow overriding. Just leave it as undefined for now. i.e., if there's multiple entries for the same path, it won't give an error, but behavior is undefined. This is still experimental, after all. We can try and figure out a better conflict resolution strategy separately.

aignas · 2025-06-08T11:50:52Z

Hmm, I am +1 on strictly mentioning that it is undefined then. We can leave this discussion about how to override targets separately.

I'll cleanup the docs as part of this PR then.

aignas · 2025-06-08T13:00:19Z

OK, PTAL, I think the majority of the stuff should be ready to merge.

python/private/py_library.bzl

tests/venv_site_packages_libs/depset_tests.bzl

rickeylev · 2025-06-09T16:21:20Z

should RECORD files be excluded

I don't think it matters? The RECORD file is just a list of files and their hashes. Bazel is already tracking all the files and their hashes. If any change, then RECORD changes, but also Bazel will see them change directly.

arrdem · 2025-06-10T00:03:49Z

+1, the RECORD file is stable, is part of the distribution, would be expected in a venv built with any other tool and should be preserved here.

aignas · 2025-06-10T08:39:23Z

+1, the RECORD file is stable, is part of the distribution, would be expected in a venv built with any other tool and should be preserved here.

@arrdem, I think we need to make an additional change in the whl_library_targets for that, so I'll leave this out of scope for this PR. Maybe we can do this only for whl_library instances that are created from whls and not from sdists, because I can see how there could be non-reproducibility for sdists based on some environment.

aignas force-pushed the exp/distinfo-venv branch from ebed371 to 9dd7589 Compare May 28, 2025 14:16

aignas changed the title ~~wip: dist-info folders in the venv~~ feat: dist-info folders in the venv May 28, 2025

aignas changed the title ~~feat: dist-info folders in the venv~~ feat: data files in the venv May 28, 2025

aignas added 3 commits May 29, 2025 22:44

Merge branch 'main' into exp/distinfo-venv

1381bf8

fixup

42e5059

add a failing test

d59f85a

add a test for topological ordering preference

714b245

aignas commented May 30, 2025

View reviewed changes

python/private/py_executable.bzl Outdated Show resolved Hide resolved

aignas added 3 commits May 30, 2025 17:39

correctly pop existing links for package overrides

31e16ee

finish impl

e0affec

cleanup stale docs

c844aa0

aignas marked this pull request as ready for review May 30, 2025 09:00

aignas requested a review from rickeylev as a code owner May 30, 2025 09:00

aignas changed the title ~~feat: data files in the venv~~ feat: data and pyi files in the venv May 30, 2025

rickeylev reviewed May 30, 2025

View reviewed changes

aignas added 9 commits May 31, 2025 15:58

add an .so test file

f726341

comment: rename src to package

ef8114e

comment: add an explanation

fa1249a

comment: pass pkg with version

30c1d01

comment: use package as the primary key

dfa24b6

a little bit of cleanup

6405e4c

more robust skipping of the dist_info

4d73e04

make the algo more robust

0c594e5

make the linking algo a little bit more robust by adding a test for e…

c70f05a

…xternal data inclusion.

aignas requested review from rickeylev and groodt May 31, 2025 10:49

rickeylev requested changes Jun 2, 2025

View reviewed changes

add depset characterization tests

d1142be

aignas marked this pull request as ready for review June 6, 2025 14:07

add all order tests for thinking

c298f3c

aignas commented Jun 6, 2025

View reviewed changes

aignas added 6 commits June 7, 2025 17:05

Reapply "Reapply "works: put back the sorting""

7ec284b

This reverts commit 507985b.

implement the sorting of targets manually

4f4e5fb

fix typo

fbcbe6d

more notes

6cfa441

docs

2081232

refactor

cdd6105

aignas added 2 commits June 8, 2025 21:56

start cleanup

c1abbeb

remove outdated docs

65632f8

rickeylev approved these changes Jun 9, 2025

View reviewed changes

python/private/py_library.bzl Outdated Show resolved Hide resolved

python/private/py_library.bzl Outdated Show resolved Hide resolved

tests/venv_site_packages_libs/depset_tests.bzl Outdated Show resolved Hide resolved

aignas added 2 commits June 10, 2025 17:39

remove depset tests

0c4dacd

update docs

eac2bba

aignas enabled auto-merge June 10, 2025 08:42

aignas added this pull request to the merge queue Jun 10, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 10, 2025

aignas added this pull request to the merge queue Jun 10, 2025

Merged via the queue into bazel-contrib:main with commit 013acd9 Jun 10, 2025
3 checks passed

aignas deleted the exp/distinfo-venv branch June 11, 2025 05:35

Uh oh!

feat: data and pyi files in the venv #2936

feat: data and pyi files in the venv #2936

Uh oh!

Conversation

aignas commented May 27, 2025 • edited by rickeylev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented May 28, 2025 • edited by rickeylev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented May 30, 2025

Uh oh!

Uh oh!

rickeylev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickeylev Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickeylev Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

aignas Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

aignas Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aignas commented Jun 5, 2025

Uh oh!

aignas Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

aignas commented Jun 7, 2025

Uh oh!

rickeylev commented Jun 7, 2025

Uh oh!

aignas commented Jun 8, 2025

Uh oh!

aignas commented Jun 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rickeylev commented Jun 9, 2025

Uh oh!

arrdem commented Jun 10, 2025

Uh oh!

aignas commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aignas commented May 27, 2025 •

edited by rickeylev

Loading

aignas commented May 28, 2025 •

edited by rickeylev

Loading

aignas Jun 4, 2025 •

edited

Loading