Thanks to visit codestin.com
Credit goes to github.com

Skip to content

compile_pip_requirements .update does not work with local sub modules #2733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
faximan opened this issue Apr 3, 2025 · 3 comments
Open

compile_pip_requirements .update does not work with local sub modules #2733

faximan opened this issue Apr 3, 2025 · 3 comments
Labels
P4 type: pip pip/pypi integration

Comments

@faximan
Copy link

faximan commented Apr 3, 2025

🐞 bug report

Affected Rule

The issue is caused by the rule: compile_pip_requirements

Is this a regression?

No

Description

When using a nested Bazel submodule, compile_pip_requirements fails to find and write requirements.txt as the path component containing the module is stripped out.

https://github.com/bazel-contrib/rules_python/pull/1124/files#diff-92abbbd273b0a63585ad40be538589b70bf6468143f0cf37152fc89b28afc526 added most of this logic (follow the resolved_requirements_txt var).

🔬 Minimal Reproduction

https://github.com/faximan/requirements-repro (see README.md)

🔥 Exception or Error


bazel --bazelrc=/dev/null run @subrepo//:requirements.update
Updating requirements.txt
Traceback (most recent call last):
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/_main/../rules_python+/python/private/pypi/dependency_resolver/dependency_resolver.py", line 238, in 
    main()
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/_main/../rules_python+/python/private/pypi/dependency_resolver/dependency_resolver.py", line 176, in main
    os.makedirs(os.path.dirname(requirements_file_relative), exist_ok=True)
  File "", line 225, in makedirs
FileNotFoundError: [Errno 2] No such file or directory: ''

🌍 Your Environment

Operating System:

  
Linux

Output of bazel version:

  
8.1.1

Rules_python version:

  
1.3.0

Anything else relevant?
faximan#1 fixes the issue for me, but probably in the wrong way. I don't think writing directly to the runfiles tree is necessarily the right thing to do? It also currently leaves behind some .__atomic-write files.

Since I am not familiar with this codebase, behavior on Windows etc. I'm looking for someone else to take this over and propose a PR.
@aignas
Copy link
Collaborator

aignas commented Apr 5, 2025

Having gone through designing the new lock rule that will be available to users in the next release (#2657) I would say that supporting the use case reliably and covering all edge cases is somewhat hard.

This does not work in:

  • the case where you want to write the file for the first time.
  • the case where the submodule is somewhere deeper - bazel just does not give you the actual path that you need to use here.

Hence I am inclined to close this as won't do unless there is a very compelling reason why not to. Just so that I know exactly the use case that you have, could you please explain in more detail what kind of workflows this would enable?

@aignas aignas added P4 type: pip pip/pypi integration labels Apr 5, 2025
@faximan
Copy link
Author

faximan commented Apr 30, 2025

Sorry, I did not see your comment until now. Thanks for having a look!

Our use case is granted somewhat odd: we have a monorepo setup with a root bazel module. Then we have a subfolder which is authored outside of the monorepo in an environment that does not have Bazel, but regularly imported into the monorepo. This subfolder is its own bazel module.

Users of the monorepo often need to build targets inside the subrepo, but they almost always do so from the root (bazel build @subrepo//my:target), so this is what Bazel will have cached, what is tested in CI etc. The requirements targets happen to live in the subrepo, but now requires cd subrepo; bazel run //:requirements.update which is then suboptimal.

Re your edge cases: our team does not care about writing the file for the first time or deeper nesting. It is probably not a compelling of a use case that you're interested in, but I think it is still a valid FR (especially the error msg is unclear).

@rickeylev
Copy link
Collaborator

Quick drive by comment

Short version: this can't be made to generally work because a submodule may not be writable.

Long version: to make it work, a client-local path has to be passed into the pip_compile rule somehow. Not sure how that could be done in a general way. Maybe tricks during the bzlmod phase with a custom extension? Not sure.

Re: run files modification: yeah, that isn't kosher. It's only working because they happen to be symlinks back to the original source, but that isn't guaranteed by bazel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P4 type: pip pip/pypi integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@faximan @aignas @rickeylev