virtual environment: either crashes or cannot find modules #1478
Comments
|
Some more random tinkering led me to a solution. The following order of operations works, deviating from the wiki:
Doing things in this order does not crash and also causes the new python path to be visible in Runtime.PythonDLL = "C:/Program Files/Python39/python39.dll";
string pathToVirtualEnv = "C:/Users/felk/venv39";
string path = Environment.GetEnvironmentVariable("PATH")!.TrimEnd(Path.PathSeparator);
path = string.IsNullOrEmpty(path) ? pathToVirtualEnv : path + Path.PathSeparator + pathToVirtualEnv;
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToVirtualEnv, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONPATH",
$"{pathToVirtualEnv}/Lib/site-packages{Path.PathSeparator}" +
$"{pathToVirtualEnv}/Lib{Path.PathSeparator}", EnvironmentVariableTarget.Process);
PythonEngine.PythonPath = PythonEngine.PythonPath + Path.PathSeparator +
Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Process);
PythonEngine.PythonHome = pathToVirtualEnv;
PythonEngine.Initialize();
dynamic sys = Py.Import("sys");
Console.WriteLine(sys.path);
Console.WriteLine(PythonEngine.PythonPath);
PythonEngine.BeginAllowThreads();
// ... any further code wrapped in using(Py.GIL()) { ... }console output: This seems to work for me, but I'll leave this issue open for now in case anyone wants to adjust the wiki. |
|
I am appending my own troubles with virtual environments to this open issue rather than create a new one. Environment
IssueI spent better part of 2 days trying to get virtual environments working with pythonnet. Part of the problem is that there are a number of various methods to create virtual environments and they may end up creating slightly different "flavored" environments. For example, see: This does not even cover conda environments which are totally separate. In order to better understand how virtual environments can be made to work with pythonnet, you have to dig into the details of how Python sets up sys.path and how this behaves with virtual environments, in particular, PEP 405 "lightweight" environments such as those created with the venv module. Key Reference Material:
If one reads through these documents, one finds that all things begin with sys.executable. Python walks up the path looking for its system libraries based on this location except in the case where it finds the file "pyvenv.cfg" one level above the executable. This is the extension provided by PEP 405 to enable lightweight virtual environments. In this case, the pyvenv.cfg "home" key tells Python where the base install is. This already presents a problem, because let's see how sys.executable gets set from a C# program using pythonnet to embed Python. Here is a test program to expose some details of the resulting Python environment. using System;
using System.Collections.Generic;
using Python.Runtime;
namespace TestPythonnet
{
class Program
{
static void Main(string[] args)
{
// using my base python install, not a venv
var pathToBaseEnv = @"C:\Users\myuser\AppData\Local\Programs\Python\Python38";
Environment.SetEnvironmentVariable("PYTHONHOME", pathToBaseEnv, EnvironmentVariableTarget.Process);
Runtime.PythonDLL = pathToBaseEnv + @"\python38.dll";
PythonEngine.Initialize();
using (Py.GIL())
{
dynamic sys = Py.Import("sys");
dynamic os = Py.Import("os");
Console.WriteLine($"PYTHONHOME: {os.getenv("PYTHONHOME")}");
Console.WriteLine($"PYTHONPATH: {os.getenv("PYTHONPATH")}");
Console.WriteLine($"sys.executable: {sys.executable}");
Console.WriteLine($"sys.prefix: {sys.prefix}");
Console.WriteLine($"sys.base_prefix: {sys.base_prefix}");
Console.WriteLine($"sys.exec_prefix: {sys.exec_prefix}");
Console.WriteLine($"sys.base_exec_prefix: {sys.base_exec_prefix}");
Console.WriteLine("sys.path:");
foreach (var p in sys.path)
{
Console.WriteLine(p);
}
Console.WriteLine();
}
PythonEngine.Shutdown();
}
}
}There are two major points to notice here:
The first issue may be related to this post regarding SetEnvironmentVariable and setenv/getenv. I have not confirmed. Regardless, it seems that whatever environment settings one makes in code do not end up in the Python environment variables. Therefore, the environment manipulation in the example code is a red herring, except for the PATH variable, which would allow one to set the Python DLL name without a full absolute path if the PATH includes the Python base folder (with pythonXX.dll). One should instead do such environment manipulations through the Run/Test environment, i.e. Visual Studio project setup for the debugging environment. Regarding the second issue, one may well ask how this example succeeded if Python does not see the PYTHONHOME variable and the system libraries are not to be found along the path to sys.executable. Examining the Stack Overflow we find:
For sys.path
Finally
Examining my registry shows that HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.8\PythonPath contains So it seems Windows was able to resolve the path issue using the registry and then applies this path to sys.prefix despite it having no resemblance to sys.executable. What about the rest of the sys.path entries?
// Need to add the runtime directory to sys.path so that we
// can find built-in assemblies like System.Data, et. al.
string rtdir = RuntimeEnvironment.GetRuntimeDirectory();
IntPtr path = PySys_GetObject("path").DangerousGetAddress();
IntPtr item = PyString_FromString(rtdir);
if (PySequence_Contains(path, item) == 0)
{
PyList_Append(new BorrowedReference(path), new BorrowedReference(item));
}
XDecref(item);
AssemblyManager.UpdatePath();So, now that we've figured out what happened, how do we fix this to work with virtual environments properly, specifically PEP 405 "lightweight" environments? Well, unfortunately, it seems we can't replicate the setup exactly without lots of manual intervention. A PEP 405 environment looks something like this (on my system): What is not included is pythonXX.dll, which is exactly the file needed by pythonnet to function. In "normal" operations, I would put Why don't we just set PYTHONHOME to the venv folder and be done with it? Unfortunately, this doesn't work because PYTHONHOME needs to point to the root of the Python install with the system libraries, which this lightweight venv does not have. (It seems PYTHONHOME is explicitly incompatible with venv because it is unset/cached in the activate script). Proposed SolutionSo, in order to replicate the venv setup, we need to follow a rather messy manual process. In the following steps, all environment variables should be set outside of code (i.e. in your shell or Visual Studio project setup) in order to avoid the problem mentioned before where values do not propagate down into Python after being changed in C#.
var pathToVirtualEnv = Environment.GetEnvironmentVariable("PYTHONNET_PYVENV"); // I invented this new enviroment variable
// or
var pathToVirtualEnv = @'C:\Users\myuser\py38'; // path to my virtualenv
// Access the API so the DLL gets loaded
if (!String.IsNullOrEmpty(pathToVirtualEnv))
{
// Access the API so the DLL gets loaded
string version = PythonEngine.Version;
// Now we can set the flag and have it stick
PythonEngine.SetNoSiteFlag();
}
PythonEngine.Initialize();
using (Py.GIL())
{
if (!String.IsNullOrEmpty(pathToVirtualEnv))
{
// fix the prefixes to point to our venv
// (This is for Windows, there may be some difference with sys.exec_prefix on other platforms)
dynamic sys = Py.Import("sys");
sys.prefix = pathToVirtualEnv;
sys.exec_prefix = pathToVirtualEnv;
dynamic site = Py.Import("site");
// This has to be overwritten because site module may already have been loaded by the interpreter (but not run yet)
site.PREFIXES = new List<PyObject> { sys.prefix, sys.exec_prefix };
// Run site path modification with tweaked prefixes
site.main();
}
}Here is a full code example. I used as much setup in the external environment as possible (because hardcoding paths into software is not a great way to go.) Environment variables (from launchSettings.json): {
"profiles": {
"TestPythonnet": {
"commandName": "Project",
"environmentVariables": {
"PYTHONNET_PYVENV": "C:\\Users\\myuser\\py38",
"PYTHONNET_PYVER": "3.8",
"PYTHONHOME": "C:\\Users\\myuser\\AppData\\Local\\Programs\\Python\\Python38",
"PATH": "C:\\Users\\myuser\\AppData\\Local\\Programs\\Python\\Python38; %PATH%"
}
}
}
}Program: using System;
using System.Collections.Generic;
using Python.Runtime;
namespace TestPythonnet
{
class Program
{
static void Main(string[] args)
{
var pathToVirtualEnv = Environment.GetEnvironmentVariable("PYTHONNET_PYVENV");
if (!String.IsNullOrEmpty(pathToVirtualEnv))
{
// Access the API so the DLL gets loaded
string version = PythonEngine.Version;
// Now we can set the flag and have it stick
PythonEngine.SetNoSiteFlag();
}
PythonEngine.Initialize();
using (Py.GIL())
{
if (!String.IsNullOrEmpty(pathToVirtualEnv))
{
// fix the prefixes to point to our venv
// (This is for Windows, there may be some difference with sys.exec_prefix on other platforms)
dynamic sys = Py.Import("sys");
sys.prefix = pathToVirtualEnv;
sys.exec_prefix = pathToVirtualEnv;
dynamic site = Py.Import("site");
// This has to be overwritten because site module may already have
// been loaded by the interpreter (but not run yet)
site.PREFIXES = new List<PyObject> { sys.prefix, sys.exec_prefix };
// Run site path modification with tweaked prefixes
site.main();
}
}
using (Py.GIL())
{
dynamic sys = Py.Import("sys");
dynamic os = Py.Import("os");
Console.WriteLine($"PYTHONHOME: {os.getenv("PYTHONHOME")}");
Console.WriteLine($"PYTHONPATH: {os.getenv("PYTHONPATH")}");
Console.WriteLine($"sys.executable: {sys.executable}");
Console.WriteLine($"sys.prefix: {sys.prefix}");
Console.WriteLine($"sys.base_prefix: {sys.base_prefix}");
Console.WriteLine($"sys.exec_prefix: {sys.exec_prefix}");
Console.WriteLine($"sys.base_exec_prefix: {sys.base_exec_prefix}");
Console.WriteLine("sys.path:");
foreach (var p in sys.path)
{
Console.WriteLine(p);
}
Console.WriteLine();
}
PythonEngine.Shutdown();
}
}
}Output: This is pretty close to the environment generated by running python normally through the venv, modulo the extra .NET folders. Proposed Enhancement RequestObviously, this is rather ugly and unfortunate. The root cause of the problem is that we cannot control the value of sys.executable so that Python can follow its standard process to load the venv. Fortunately, there is a new config API added to Python 3.8 with PEP 587. This API allows fine-grained control over the startup Python configuration, including sys.executable. Implementing access to this API through pythonnet would allow us to simply set sys.executable to the venv version of python.exe and have everything work from there. |
|
Thank you for the detailed report. PRs for this are very welcome! A few comments:
|
|
@Meisterrichter offtopic idea: if you want your .NET app run within a virtualenv, can you maybe launch it from Python executable? E.g. do # TODO: setup clr first
import clr
clr.AddReference('MyApp')
from MyApp import PythonEntryPoint
PythonEntryPoint.PublicMainFunction()? |
|
I did a hack just to get past this problem while doing development in VSCode in a Debian devcontainer on MacOS. My production environment (AWS EC2 with Amazon Linux) will not need this hack, and I don't recommend using it anywhere except getting past the issue during development. First, the above post from @bpdavis86 was incredibly helpful and informative. In my case, I could not get a venv environment working through pythonnet. I needed a way to add site-packages system-wide, but packages were maintained by apt and not pip. Running I have no idea why the zip file is in the path and have not investigated. It does not exist. The USER_SITE value led to the hack. Specifically, I first created a venv, activated it, and installed my required packages. I then created the USER_SITE folder and copied the packages to it. Running my C# app that runs a script via pythonnet worked. Again - this is just a hack. |
Environment
Details
according to #984 .NET Core is supported, which I hope includes .NET 5.0
according to #1389 Python 3.9 is supported in pythonnet 2.5, though I was unable to find 2.5 on nuget, so I went with a 3.0 preview in the hopes that it will work too.
Describe what you were trying to get done.
Embed python in a C# application with a virtual environment (venv) and import a module from it.
I have set up a virtual environment at
C:\Users\felk\venv39and verified that it works and I can use modules that are available in the venv but not globally:According to the wiki this is the code to get virtual environments working:
However it is not clear to me what comes after that, most importantly where do I call
PythonEngine.Initialize();, so I attempted to piecemeal something together:PythonEngine.Initialize();after the above snipped causes the application to crash without an exception, which I suspected is a segfault due to threading issuesPythonEngine.PythonHomeandPythonEngine.PythonPathseems to work, in a sense that the application does not crash and calls into python from C# are possible.However, when I try to import a module present in the virtual environment, it is not being found. I can observe that the venv path is part of
PythonEngine.PythonPathbut not part ofsys.pathby checking after settingPythonEngine.PythonPath:this results in
This seems to be unrelated, but I also removed this line from the wiki's example as it seems nonsensical to override the
PATHwith the venv path if it was just set to the correctly appended one a line above, and since the comment above literally just saidbe sure not to overwrite your existing "PATH" environmental variable:I also tried appending to
PythonEngine.PythonPathinstead of replacing it, as described in #1348 but that had no effect either.The text was updated successfully, but these errors were encountered: