I recall there was a change to avoid log spam spam when alternative indexes were available. I wonder if that is contributing? I think part of that fix was to change it to fetch an entire index, figure out what was left, then fetch again. The behavior previously was to immediately fetch the next "step" as soon as any given package finished. Don't quote me on this exactly, though -- Ignas would remember better.

Relevant snippet from slack:

Ignas says:
I think it could be that it is fetching more wheels, because indexes may provide multiple wheels that are compatible with target platforms. This could be improved by providing at most one wheel that is the most specialized wheel for the given platform, but then it makes things like musl harder to support.
So bazel query in your case will fetch the musl and manylinux variants of the wheel because you are using query instead of cquery.

This sounds vaguely familiar -- didn't we have a similar problem with our doc building using iblaze? bazel query was trying to fetch windows wheels and failing, but we don't care about windows. I forget what the fix was.

In any case, I think a reasonable functionality request is to provide some basic restriction capability for how it traverses the indexes. e.g. if you simply don't care about musl (or windows, or w/e), then it shouldn't try to follow any edges that are musl-specific, try to download any metadata for musl-specific wheels, etc.

(as an aside, doesn't the uv.lock format and pylock.toml have capabilities that would prevent this problem?)

(as an aside, doesn't the uv.lock format and pylock.toml have capabilities that would prevent this problem?)

At the very least there is no index fetching because uv lock encodes all possible urls in the lockfile:

[[package]]
name = "markupsafe"
version = "3.0.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b2/97/5d42485e71dfc078108a86d6de8fa46db44a1a9295e89c5d6d4a06e23a62/markupsafe-3.0.2.tar.gz", hash = "sha256:ee55d3edf80167e48ea11a923c7386f4669df67d7994554387f84e7d8b0a2bf0", size = 20537 }
wheels = [
    { url = "https://files.pythonhosted.org/packages/6b/28/bbf83e3f76936960b850435576dd5e67034e200469571be53f69174a2dfd/MarkupSafe-3.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:9025b4018f3a1314059769c7bf15441064b2207cb3f065e6ea1e7359cb46db9d", size = 14353 },
    { url = "https://files.pythonhosted.org/packages/6c/30/316d194b093cde57d448a4c3209f22e3046c5bb2fb0820b118292b334be7/MarkupSafe-3.0.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:93335ca3812df2f366e80509ae119189886b0f3c2b81325d39efdb84a1e2ae93", size = 12392 },

Could you please provide the following numbers for each of the cases:

• How many packages do you have? bazel query "@pip//..." | rg :dist_info | wc -l
• How many wheels do you have? bazel query "kind(py_library, deps(@pip//...))" | rg :pkg| wc -l
• If you first do bazel build @pip//:BUILD.bazel, do the numbers change? This should remove the step of querying the SimpleAPI from the equation.
• What happens if you do cquery instead of query? How do the numbers differ then?