pypy support in pythonnet #330
Comments
src/monoclr/pynetinit.c: In function ‘main_thread_handler’:
src/monoclr/pynetinit.c:106:5: error: unknown type name ‘Dl_info’
src/monoclr/pynetinit.c:107:5: warning: implicit declaration of function ‘dladdr’ [-Wimplicit-function-declaration]
src/monoclr/pynetinit.c:107:22: error: ‘Py_Initialize’ undeclared (first use in this function)
src/monoclr/pynetinit.c:107:22: note: each undeclared identifier is reported only once for each function it appears in
src/monoclr/pynetinit.c:109:23: error: request for member ‘dli_fname’ in something not a structure or union
src/monoclr/pynetinit.c:110:29: error: request for member ‘dli_fname’ in something not a structure or unionabove is the traceback from building on pypi/travis |
|
I tried on pypy with pythonnet on Win10. It builds fine, but then fails to find |
|
interesting. you tried building using the I've don't know if the error is the same or not in either case. |
|
can I pass debug flag to pip/setuptools (not from VS)? |
|
its on my todo list to add a |
|
still the same error, here is the full output from the build: |
|
This may have something to do w it. |
|
@vmuriart yes, that's possible that we are using some unsupported C-API features in cpyext. But it fails silently. I'm trying with nightly pypy builds: |
|
same issue on nightly pypy build: |
|
I see that it creates and installs After renaming it manually, it fails differently, which is probably another issue. |
|
@arigo thanks, so now clr is looking for python27.dll, which I guess is libpypy-c.dll: Then I created a "soft" symlink the loaded libraries look like expected:
|
|
libpypy-c.dll exposes the symbols under names |
|
OMG, now we need two sets of names for few hundred C-API pinvokes in C# code: Click to expand the search results```Find all "internal.\n.Py._", Regular expressions, Find Results 1, Current Document |
|
Yeah, it's a bit of a pain :/. We could have two |
|
Maybe a subset of runtime.cs could be auto-generated by parsing Python.h, similar to how the interopXX.cs files are generated? |
|
Note, before going to lengths in that direction, that PyPy's emulation of the CPython C extension API is not 100% complete. For example, |
|
using this regex I replaced all
I had to remove UCS2/UCS4 calling convention from Unicode objects as well. I ignored the call to Py_Initialize() for extending case with try/catch (this works in CPython), but is there any alternative method to initialize PyPy runtime when embedding it? I guess in the worst case we could resort to cffi embedding API? http://cffi.readthedocs.io/en/latest/embedding.html Anyway with extending ( |
|
@arigo are there debug builds of pypy with PDB symbols anywhere? i read it takes hours to compile pypy locally. |
|
@filmor @denfromufa this looks like a good example to use @dmitriyse 's implementation of separating Since it uses interfaces for defining runtime, we can easily create a implementation for |
|
Let me guess, if you don't read Python.h, then the .NET code also missed the fact that the base PyObject structure is different (and so, all others too). PyPy is really not meant to be an ABI-compatible replacement for CPython---only an API-compatible one. Another guess: for performance reasons the .NET code contains explicit versions of various CPython macros rewritten as .NET code. Sorry, that can't work either: either the macros are different, or they are implemented as full function calls. |
|
@arigo you are right, but pythonnet does not use too many internal CPython structures. I believe we can port all of those structures either manually or using bundled |
|
here is the pypy + pythonnet with my edits if anyone wants to play with or start a new branch: https://www.dropbox.com/s/zevjlsfe9mk1pdg/pypy2-v5.6.0-win32.zip?dl=0 |
|
@arigo @tonyroberts I think Decref/Incref and PyPyObject_Type macros are resolved. Now I'm dealing with this issue in pycparser for parsing PyHeapTypeObject and all its sub-structs: eliben/pycparser#162 (comment) If this is not resolved, then will need to regenerate PyHeapTypeObject from Python.h manually. Here is the current crash state in pypy with |
|
PyObject_HEAD contains extra
Now, next error: Here is the current state: https://www.dropbox.com/s/zevjlsfe9mk1pdg/pypy2-v5.6.0-win32.zip?dl=0 |
|
Oh, now I reached RPython traceback: |
|
Here is the current pypy branch: https://github.com/pythonnet/pythonnet/tree/pypy I'm stuck with above error from rpython with this CLR traceback:
My plan is to try out with debug build of pypy and possibly attach with native debugger. |
|
ok, I built pypy from source and now can debug the rpython translated to C. But I have no idea why the object became undefined:
|
|
the structure of CLR Metatype looks fine:
|
|
I have some progress on pypy debugging with pythonnet. The issue right now is setting tp_flags which crashes PyType_Ready: https://bitbucket.org/pypy/pypy/issues/2486/pypytype_ready-with-tp_flags-crashes-pypy some issues were resolved in pypy by @arigo, which enable parsing pyheaptypeobject without hacking the source: https://bitbucket.org/pypy/pypy/issues/2484/pyheaptypeobject-difference-with-cpython https://bitbucket.org/pypy/pypy/issues/2483/clang-complains-about-parsing-pypy-headers https://bitbucket.org/pypy/pypy/issues/2478/sysconfigget_config_var-includepy-empty-on The only difference in pyheaptypeobject is still ob_pypy_link like I pointed out above by using diff. |
|
some progress in these issues: https://bitbucket.org/pypy/pypy/issues/2486/pypytype_ready-with-tp_flags-crashes-pypy I can actually do |
|
One more issue to track in PyPy cpyext: https://bitbucket.org/pypy/pypy/issues/2447/cpyext-_pyobject_getdictptr-is |
|
any hope ? |
|
I'm personally waiting for hpy to mature before working on this myself. If anyone wants to take another shot at cpyext (which should be really easy now!), I'll gladly help out. |
|
Conda-forge recently started the support for PyPy on Windows and started a migration. As I can read here I can close this Merge Request for now? |
|
Yes. There is no support for Pypy in Python.NET. |
|
Can it be achieved? I also need to use pythonnet in pypy now.
|
|
I also would like to see this. Has there been any progress? |
|
It seems like there's no way. |
|
I think the whole approach is doomed for PyPy or any non-CPython interpreter. A way forward might be to use CFFI embeddings, like mentioned above: you can build with CFFI a DLL that basically exports whatever (C-level) API you like, and then you can write C# code to call functions in that DLL, install callbacks, and so on. You don't get any of the pythonnet-like features, though; only a C API. Depending on your goal, it might be enough. Alternatively, someone could then write a C API that gives Python some generic control over the .NET objects, with logic both on the C# and the Python side to give generic wrappers of C# objects in Python and/or vice-versa, using runtime introspection on both sides. |
|
I think the most important reason for choosing Pypy is to consider runtime performance, but currently I think it may not be necessary because Python 3.13 has already removed GIL, so I may not prioritize choosing Pypy. |
cpyext is mature enough (numpy, cython, scipy, pandas soon) to support pythonnet in pypy. In fact pypy used to have a branch which ran on top of CLR (similar to IronPython), but this branch is not maintained.
The use of pypy in .NET is quite popular request:
http://stackoverflow.com/questions/6084697/is-it-possible-to-embed-pypy-into-a-net-application
The only limitation of pypy is 32-bit on Windows and poor support for Python 3+, although the latter is improving with funding from Mozilla.
Also this might be quite nice collaboration with pypy developers due to their level of expertise in python C-API and cffi.
The text was updated successfully, but these errors were encountered: