Just thinking out loud about alternatives here.

Recently we have merge #2507, which is a single target where the user get the interpreter for a particular target by:

bazel run @rules_python//python/bin:python --@rules_python//python/config_settings:interpreter_target=<my_target>

I think it may be still missing the dependencies propagated into the interpreter environment but that was something that still needs to be done and probably is not a super heavy lift. @philsc am I right here?

Env vars that in the long term may become a little bit hard to maintain and usually are a leaky abstraction. Using the interpreter target from above would allow one to just pass the args as is to the bazel run command. It does look awfully similar to what you want to build.

In the command:

--file /path/to/built/python-file

Is this really mandatory? What is the /path/to/build/python-file really used for? If the python files where somehow present in the PYTHONPATH, would it be enough?

Hi @aignas;

where the user get the interpreter for a particular target by:

Unless I am misunderstanding, I am unsure that this would help because the build product is still a bash script and not a Python script.

Is this really mandatory?

It acts as an argument to the pydevd.py program instructing it which secondary Python program to run instrumented / augmented with the debug tool.

where the user get the interpreter for a particular target by:

Unless I am misunderstanding, I am unsure that this would help because the build product is still a bash script and not a Python script.

I was thinking that the python3 in your command above has to be the interpreter from within the bazel target, but it can be the local env. I thought that there is a need to have all of the PYTHONPATH propagated into the script because the intelij plugin needs to know where the dependencies are or what they are. I think I was mistaken on these points.

Is this really mandatory?

It acts as an argument to the pydevd.py program instructing it which secondary Python program to run instrumented / augmented with the debug tool.

What does the pydevd.py do? Is it for coverage, debugging or LSP stuff? Sorry if I am digging too deep into the specifics, but I want to avoid the XY problem. I want to understand what we are really trying to solve here.

I'm ok with an env var for this. Just unset it after bootstrapping so it doesn't get inherited by subprocesses.

A few questions/comments:

Specifying a path to the new entry point: IMHO, we should allow both an arbitrary path and a runfiles-root relative path. My rationale is an arbitrary local path is handy if I just want to use a debugger I've installed locally. This will be inherently prone to problems, but that's on the user; this is just for debugging, after all.

When to invoke the debugger: possible alternative? instead of python3 mydebugger.py <rest of args>, another option might be to have the bootstrap invoke the debugger. i.e, the invocation is more like python3 --RPXrules_python_debugger=mydebugger:main <rest of args>. I propose this because, until stage2 sets up sys.path, I'm not sure the debugger's main will be reliably usable. It would have to be a self-contained file (only stdlib allowed). "mydebugger" here could be a module, which might be a better abstraction.

I think I was mistaken on these points. ... What does the pydevd.py do?

When I mean plugin, I am referring to this.

The IDE requires an SDK Python interpreter to function. This is not necessarily the exact same one present in the sandbox; it would be an external venv or a platform-supplied interpreter for example. In the code-snippet in the description this is the python3 binary. When the IDE runs the debugger, it will use this SDK interpreter to launch the rules_python bootstrap system.

The pydevd.py is a small self-contained Python module / program which instruments the Python environment for debugging and then invokes the main program which is identified by the --file argument. In a way, it intercepts the main program's startup and orchestrates the debugging before starting the main program.

I've always felt that the IDE plugin should probably integrate with debuggers differently. It does seem fragile to require or expect a "system" python to be able to run or debug a "bazel-managed py_binary". I understand why they did it this way, but I don't think anyone ever spoke to anyone from rules_python about how best to integrate :) so it's more an accident that it works at all, than by design.

Just unset it after bootstrapping so it doesn't get inherited by subprocesses.

Good call; I have just added that unset.

When to invoke the debugger: possible alternative? instead of python3 mydebugger.py , another option ...

If I understand correctly, I am seeing this alternative has an inverse problem in that the runtime Bazel execution environment would then be "polluted" by debug dependencies and from the perspective of the IDE, the Bazel target would need to be explicitly setup to have those dependencies to be able to be debugged rather than "you can debug any py_binary or py_test target".

By observation only, it seems like the pydevd approach above is self-contained and the debugpy approach appears similar (see here) so I am thinking that the env-var approach in this PR seems aligned with their expected usage patterns.

I should point out though that no work has been done on making this function with any IDE yet; this is anticipatory.

It does seem fragile to require or expect a "system" python to be able to run or debug a "bazel-managed py_binary"

This seems to be the case today, but as I understand it, if the IDE ends up running stage1_bootstrap_template.sh as the entry-point then it should be running the debugger with the correct interpreter?

This seems to be the case today, but as I understand it, if the IDE ends up running stage1_bootstrap_template.sh as the entry-point then it should be running the debugger with the correct interpreter?

Indeed. But that seems like coupling to implementation details. It might be better for rules_python targets to expose configuration or similar that IDE can than rely on as an agreed interface, rather than IDE sniffing around inside the implementation. If the IDE are coupling to stage1_bootstrap_template.sh it causes breakage later:

• what if this bootstrap is removed?
• what if this bootstrap is renamed?

It might be better for rules_python targets to expose configuration or similar that IDE can than rely on as an agreed interface

Yes, true; prior to the bash bootloader this was probably not unreasonable actually, but you're right that with the introduction of the bash-based bootloader, a future IDE integration would be making specific assumptions about how the bootloaders are structured so it can manifest different behaviour for bash and Python.

Moving the logic for initializing the debugger into the bootloader gets a bit messy though as there's the two main debugger integrations (that I am aware of) which would be initialized in different ways, each may have their own parameters, each may be at different version, each may be version-coupled to an IDE version (eg here) and its probably not great to have their libraries being part of the Bazel build product either.

In balance, I still think this env-var style approach is the better one, but perhaps the semantics about how it should work in relation to the interpreter and the target Python executable script would be good to define in documentation as well. WDYT?