When statically parsing pyproject.toml, convert relative paths to absolute for use as file URIs #2221

sirosen · 2025-08-04T17:02:12Z

When statically parsing a pyproject.toml, we can encounter self-referential package usage. For example, an extra which names another extra on the same package.

The source for these requirements is the current package, but as a concrete requirement, not an abstract one. To make it concrete, we use a URI dependency link pointed at the package directory as a file-URI.

When the input path to pyproject.toml is relative, it needs to be made absolute for use as a URI. Failing to do so results in errors.

Because the pip-compile output of an such an extra depends concretely on the specific version of the package found in-tree, and not abstractly on the package name (without the URI), tests confirm that we see the rendered package directory as a file URI in pip-compile output.

Contributor checklist

Included tests for the changes.
A change note is created in changelog.d/ (see changelog.d/README.md for instructions) or the PR text says "no changelog needed".

Maintainer checklist

If no changelog is needed, apply the skip-changelog label.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

changelog.d/2215.bugfix.md

tests/test_build.py

tests/test_cli_compile.py

webknjaz · 2025-08-04T19:53:53Z

tests/test_cli_compile.py

+    assert out.exit_code == 0
+    assert out.stdout == dedent(
+        f"""\
+        foo[ext1] @ {src_file.parent.absolute().as_uri()}


Is this really what we want? I don't think people would expect absolute paths from someone else's computer to end up in their lock files..

I agree, and it's why I was pretty hesitant at first. But also, that's the behavior from 7.4.1 -- it's just not tested.

In theory, someone could be using this in some controlled context like an image build and wanting the concrete dependency path that this provides. So although I'm uncertain that it's right for the long term, I don't want to treat it as blocking the fix.

Wait, did it always use absolute paths in 7.4.1? Not relative? I thought it'd conform to the shape of input that the 7.5.0 changelog promises.. Can we use file://./rel/path/?

The path in this case is part of the InstallRequirement itself -- the "link" attribute is populated with this file URI.
The normalization in 7.5.0 only applies to the comes_from data ("via").

We could reach down into this as well and convert to file://./rel/path/ as you say, but I'm not sure if we should. It would be a separate bit of data transformation -- we could share code but it's a different attribute of the parsed requirements data that we'd be manipulating.

I suppose, we could postpone that for now. There might be a use case coverage missing for stripping the @ http://..source in some contexts.

webknjaz · 2025-08-04T22:38:26Z

Oh, looks like Windows is unhappy about the delimiters again.

sirosen · 2025-08-05T01:32:28Z

The thing I broke in the tests is treatment of comes_from data in this scenario. We're stringifying the path to the package dir here:

pip-tools/piptools/build.py

Line 91 in 6e0ef33

comes_from = f"{package_name} ({src_file})"

It's easy to "fix" by just adding an as_posix() call there, but I can imagine this becoming inconsistent with the behavior when that information is rendered via some other path. I'm going to run some experiments later this week to try to figure out what the variations are on this.

# Pull Request There is an issue with self-referential packages in a pyproject.toml with the latest pip-tools release. The self-referential [all] package, which is not written to the requirements.txt as it is not selected still throws a pip-compile error. More information for this can be found in the ticket: jazzband/pip-tools#2215 The pin should be removed when jazzband/pip-tools#2221 gets merged.

sirosen · 2025-08-13T14:33:13Z

I've just gotten back to this, taken a look at the open threads and marked the ones which were applied as suggestions as resolved.

I looked at what happens in the case of dynamic metadata, and we have a similar bit of path formatting in there.

I'd like to resolve this by adding calls to as_posix() to both locations in piptools/build.py.
One is the place mentioned above, where a static build is happening and we're writing the comes_from data.
The other is this line in this function which does the same work on requirements from dynamic metadata.

@webknjaz, are you okay with that increase in scope? I'll need to add some new tests to cover it and another changelog note.
As an overall project direction, I think it's best to try to make all of our outputs posix paths, to make the outputs consistent.

webknjaz · 2025-08-19T14:40:33Z

@sirosen agreed. Increasing the scope sounds justified.

sirosen · 2025-09-24T19:52:03Z

I've rebased to keep this fresh with the latest fixes in main and added a new changelog note changelog.d/2221.bugfix.md to cover the new path normalization which happens.

With the click-related issue starting to impact users -- and now that I'm clear that it's not a click bug but an intentional change in behavior -- I'm wanting to push this forward and try to narrow what we put into 7.5.1 .

sirosen · 2025-09-25T21:13:53Z

I think something is wrong with the codecov integration.
The report on uncovered lines doesn't make much sense:
https://app.codecov.io/gh/jazzband/pip-tools/pull/2221?dropdown=coverage

One thing I notice is that it seems to mark everything related to piplowest testing as uncovered, which suggests that somehow we're not producing distinct coverage data for those builds in a way that codecov understands.

I don't think it's at all related to this PR.

webknjaz · 2025-09-26T01:46:43Z

@sirosen will this address #2215 / #2233 ?

webknjaz · 2025-09-26T01:48:48Z

changelog.d/2215.bugfix.md

Should we symlink 2233 (upon confirmation), also 2221 (or 2221.bugfix.1.md rather)?

Yeah, #2233 is getting fixed here as well, so I added the symlink. And I did the 2221.bugfix.1.md one as well, since "why not?" 😄

I'm going to put this in the merge queue, along with any other changes I can round up right now.

@sirosen the issues didn't end up auto-closed. Next time, plz try to remember to mark the PR as closing them (via web UI or closes/resolves/fixes keywords).

webknjaz

Looks good, but maybe add those change note refs before merging..

When statically parsing a pyproject.toml , we can encounter self-referential package usage. For example, an extra which names another extra on the same package. The source for these requirements is the current package, but as a concrete requirement, not an abstract one. To make it concrete, we use a URI dependency link pointed at the package directory as a file-URI. When the input path to `pyproject.toml` is relative, it needs to be made absolute for use as a URI. Failing to do so results in errors. Because the `pip-compile` output of an such an extra depends concretely on the specific version of the package found in-tree, and not abstractly on the package name (without the URI), tests confirm that we see the rendered package directory as a file URI in `pip-compile` output.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

When recording requirements derived from pyproject.toml data, the `comes_from` field includes the path to the pyprojct.toml file. Because this path is part of the output, which should therefore be stable across `pip-tools` invocations on different platforms, normalize these paths to posix-style.

psf-chronographer bot added the bot:chronographer:provided label Aug 4, 2025

sirosen added this to the 7.5.1 milestone Aug 4, 2025

webknjaz reviewed Aug 4, 2025

View reviewed changes

sirosen force-pushed the fix-relpath-static-pyproject-parse branch 2 times, most recently from fc92905 to b05d736 Compare August 4, 2025 23:55

webknjaz moved this to 🧐 @webknjaz's review queue 📋 in 📅 Procrastinating in public Aug 7, 2025

webknjaz added this to 📅 Procrastinating in public Aug 7, 2025

This comment was marked as resolved.

Sign in to view

This comment was marked as off-topic.

Sign in to view

maurerle mentioned this pull request Aug 13, 2025

docker: fix build by pinning pip-tools assume-framework/assume#626

Merged

This comment was marked as off-topic.

Sign in to view

sirosen force-pushed the fix-relpath-static-pyproject-parse branch from 10d5cf9 to d42c30b Compare September 16, 2025 03:28

sirosen force-pushed the fix-relpath-static-pyproject-parse branch from d42c30b to bac58b8 Compare September 24, 2025 19:49

sirosen force-pushed the fix-relpath-static-pyproject-parse branch from bac58b8 to 54ced7c Compare September 25, 2025 21:23

webknjaz reviewed Sep 26, 2025

View reviewed changes

webknjaz approved these changes Sep 26, 2025

View reviewed changes

sirosen and others added 4 commits September 26, 2025 09:51

Fix formatting and provide test parameter ids

11af7b5

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

Link second resolved issue in changelog

a5cf40c

sirosen force-pushed the fix-relpath-static-pyproject-parse branch from 23b4447 to a5cf40c Compare September 26, 2025 14:51

sirosen enabled auto-merge September 26, 2025 14:51

sirosen added this pull request to the merge queue Sep 26, 2025

Merged via the queue into jazzband:main with commit 16c793b Sep 26, 2025
45 checks passed

github-project-automation bot moved this from 🧐 @webknjaz's review queue 📋 to 🌈 Done 🦄 in 📅 Procrastinating in public Sep 26, 2025

sirosen deleted the fix-relpath-static-pyproject-parse branch September 26, 2025 15:24

This was linked to issues Sep 26, 2025

Errors with referential optional-dependencies on 7.5.0 #2233

Closed

pip-compile fails to parse a pyproject.toml with self-references when supplied as a relative path #2215

Closed

This was referenced Sep 26, 2025

pip-compile fails to parse a pyproject.toml with self-references when supplied as a relative path #2215

Closed

Errors with referential optional-dependencies on 7.5.0 #2233

Closed

Uh oh!

When statically parsing pyproject.toml, convert relative paths to absolute for use as file URIs #2221

When statically parsing pyproject.toml, convert relative paths to absolute for use as file URIs #2221

Uh oh!

Conversation

sirosen commented Aug 4, 2025

Contributor checklist

Maintainer checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz commented Aug 4, 2025

Uh oh!

sirosen commented Aug 5, 2025

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as off-topic.

This comment was marked as off-topic.

sirosen commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webknjaz commented Aug 19, 2025

Uh oh!

sirosen commented Sep 24, 2025

Uh oh!

sirosen commented Sep 25, 2025

Uh oh!

webknjaz commented Sep 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sirosen commented Aug 13, 2025 •

edited

Loading