Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Delegates declared using NewReference and BorrowedReference structs are not working/crashing on ppc64le/s390x #2186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alhad-deshpande opened this issue Jul 6, 2023 · 10 comments
Open

Delegates declared using NewReference and BorrowedReference structs are not working/crashing on ppc64le/s390x #2186

alhad-deshpande opened this issue Jul 6, 2023 · 10 comments
Labels
not planned - PRs welcome

Comments

@alhad-deshpande
Copy link

Environment

  • Pythonnet version: 3.1.0-preview2023-04-03
  • Python version: 3.8.10
  • Operating System: Linux (ppc64le/s390x architecture)
  • .NET Runtime: net7.0

Details

The delegates declared using NewReference or BorrowedReference structs are not working/crashing on ppc64le architecture as the delegate declaration and C function declaration are not matching. For e.g.:
unsafe internal static delegate* unmanaged[Cdecl]<long, NewReference> PyLong_FromLongLong { get; }

The above delegate does not work because the long value is getting passed in wrong register as NewReference is a struct type but the C function just returns a pointer to PyObject. Following is the C function declaration:
PyObject* PyLong_FromLongLong(long long v)

If we change the above delegate as below then it's works fine.
unsafe internal static delegate* unmanaged[Cdecl]<long, IntPtr> PyLong_FromLongLong { get; }

    print('TODO')
  • If there was a crash, please include the traceback here.

Below is the Traceback:

Thread 1 (Thread 0x7603109c5400 (LWP 919128)):
#0 0x000076031036f318 in __GI___wait4 (pid=, stat_loc=0x7fffe1938990, options=, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1 0x000076031036f20c in __GI___waitpid (pid=, stat_loc=, options=) at waitpid.c:38
#2 0x0000760310014abc in dump_native_stacktrace (signal=, mctx=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/mini-posix.c:843
#3 mono_dump_native_crash_info (signal=, mctx=0x7fffe19394a8, info=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/mini-posix.c:870
#4 0x000076030ffc474c in mono_handle_native_crash (signal=0x7603100f2f5f "SIGSEGV", mctx=0x7fffe19394a8, info=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/mini-exceptions.c:3005
#5 0x000076030ff0e0a8 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fffe193a4e8, context=0x7fffe1939770, debug_fault_addr=0xea6100f8ea4100f0) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/mini-runtime.c:3773
#6
#7 _Py_INCREF (op=0xea6100f8ea4100f0) at ../Include/object.h:459
#8 insertdict (mp=mp@entry=0x76030c096590, key=key@entry=0x760305e0d040, hash=-8286388524546298114, value=value@entry=0xea6100f8ea4100f0) at ../Objects/dictobject.c:1034
#9 0x0000760305f34060 in PyDict_SetItem (op=0x76030c096590, key=0x760305e0d040, value=0xea6100f8ea4100f0) at ../Objects/dictobject.c:1545
#10 0x0000760305f390f4 in PyDict_SetItemString (v=0x76030c096590, key=, item=0xea6100f8ea4100f0) at ../Objects/dictobject.c:3417
#11 0x00007603060c5bc0 in PySys_SetObject (name=, v=) at ../Python/sysmodule.c:112
#12 0x000076030f525a14 in ?? ()
#13 0x000076030f524470 in ?? ()
#14 0x000076030f523ca4 in ?? ()
#15 0x000076030f511aac in ?? ()
#16 0x000076030f4dab18 in ?? ()
#17 0x000076030f4da74c in ?? ()
#18 0x000076030f4d7fc0 in ?? ()
#19 0x000076030f4d4b90 in ?? ()
#20 0x000076030f4d7dd0 in ?? ()
#21 0x000076030ff11874 in mono_jit_runtime_invoke (method=0x38e5de557e8, obj=0x0, params=, exc=, error=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/mini-runtime.c:3570
#22 0x000076030fdeca38 in do_runtime_invoke (method=0x38e5de557e8, obj=, params=0x7fffe193b580, exc=0x0, error=0x7fffe193b628) at /root/alhad/Native_Build/runtime/src/mono/mono/metadata/object.c:2415
#23 mono_runtime_invoke_checked (method=0x38e5de557e8, obj=, params=0x7fffe193b580, error=0x7fffe193b628) at /root/alhad/Native_Build/runtime/src/mono/mono/metadata/object.c:2583
#24 0x000076030fdf63b4 in do_exec_main_checked (method=0x38e5de557e8, args=, error=0x7fffe193b628) at /root/alhad/Native_Build/runtime/src/mono/mono/metadata/object.c:4435
#25 mono_runtime_exec_main_checked (method=, args=, error=) at /root/alhad/Native_Build/runtime/src/mono/mono/metadata/object.c:4541
#26 0x000076030ff76314 in mono_jit_exec_internal (domain=, assembly=, argc=1575548528, argv=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/driver.c:1365
#27 mono_jit_exec (domain=, assembly=, argc=1575548528, argv=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/driver.c:1310
#28 0x000076030ff79eb0 in main_thread_handler (user_data=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/driver.c:1442
#29 mono_main (argc=, argv=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/driver.c:2647
#30 0x000076030fffd668 in monovm_execute_assembly (argc=0, argv=0x0, managedAssemblyPath=, exitCode=0x7fffe193bdc8) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/monovm.c:268
#31 0x000076030fefdbc8 in coreclr_execute_assembly (hostHandle=, domainId=, argc=, argv=, managedAssemblyPath=, exitCode=) at /root/alhad/Native_Build/runtime/src/mono/mono/mini/main-core.c:89
#32 0x0000760310179aa4 in ?? () from /root/alhad/Native_Build/output/.dotnet/shared/Microsoft.NETCore.App/7.0.4/libhostpolicy.so
#33 0x0000760310188654 in ?? () from /root/alhad/Native_Build/output/.dotnet/shared/Microsoft.NETCore.App/7.0.4/libhostpolicy.so
#34 0x0000760310188b80 in ?? () from /root/alhad/Native_Build/output/.dotnet/shared/Microsoft.NETCore.App/7.0.4/libhostpolicy.so
#35 0x00007603101898fc in corehost_main () from /root/alhad/Native_Build/output/.dotnet/shared/Microsoft.NETCore.App/7.0.4/libhostpolicy.so
#36 0x000076031020385c in ?? () from /root/alhad/Native_Build/output/.dotnet/host/fxr/7.0.4/libhostfxr.so
#37 0x0000760310202310 in ?? () from /root/alhad/Native_Build/output/.dotnet/host/fxr/7.0.4/libhostfxr.so
#38 0x00007603101fdc20 in hostfxr_main_startupinfo () from /root/alhad/Native_Build/output/.dotnet/host/fxr/7.0.4/libhostfxr.so
#39 0x0000038e23a41d68 in ?? ()
#40 0x0000038e23a421b8 in ?? ()
#41 0x0000760310294ccc in generic_start_main (main=0x38e23a420f0, argc=, argv=0x7fffe193cb48, auxvec=0x7fffe193cc18, init=, rtld_fini=, stack_end=, fini=) at ../csu/libc-start.c:308
#42 0x0000760310294ea4 in __libc_start_main (argc=, argv=, ev=, auxvec=, rtld_fini=, stinfo=, stack_on_entry=) at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:98
#43 0x0000000000000000 in ?? ()
[Inferior 1 (process 919128) detached]

    print('TODO')
@alhad-deshpande
Copy link
Author

This is due to the type mismatch - where the C side uses a plain pointer, while the C# side uses a struct containing a pointer and makes the assumption that these two types use the same calling convention. This is true on some, but not all platform ABIs.

@lostmsu
Copy link
Member

lostmsu commented Jul 6, 2023 •
edited
Loading

This might be a bug in .NET runtime on the corresponding platform.

Anyway without the access to the platform we can not investigate. And there's no plan to officially support the architecture.

@uweigand
Copy link

uweigand commented Jul 6, 2023

No, it's not a bug in the .NET runtime. (If it were, I'd be happy to fix it in the runtime.) It's a bug in how delegates are declared in Python.NET code. The type needs to match the type that is actually being used by the target C routine, or else the runtime cannot interface correctly with native code.

Here, the C# code in Python.NET code declares a delegate that specifies that the target C routine uses a "struct" containing a single IntPtr member as return value. However, the actual C routine doesn't use a struct type as return type, but a plain pointer type.

For example, the PyLong_FromLongLong routine is defined as:

PyObject *PyLong_FromLongLong(long long ival);

In the s390x ABI, this function receives the ival parameter in register 2, and returns its result also in register 2.

However, the C# runtime thinks (because the delegate type was declared that way), that it is calling a function defined as:

struct { intptr_t ptr } PyLong_FromLongLong(long long ival);

And that function, in the s390x ABI, receives its ival parameter in register 3, and receives in register 2 the address of a buffer where the return value needs to be stored. So this is how the C# runtime sets up registers before calling native C code, which then crashes because it expects them to be set up differently, as described above.

The current Python.NET code happens to work on Intel and some other platforms, because there the ABI defines that a struct { intptr_t ptr } is returned the same way as a plain intptr_t (which in turn is returned the same way as the actual type, PyObject *). But that is simply an accidental property of those particular platforms, it isn't guaranteed to hold true everywhere (and does in fact not hold true everywhere).

The only way to fix this is to use the correct type in the delegate declaration on the C# side (in this case IntPtr - which does match the calling convention for pointer types on all platforms), and update the callers of the delegate routines accordingly (e.g. wrapping that returned IntPtr into a NewReference if required).

@lostmsu
Copy link
Member

lostmsu commented Jul 6, 2023

@uweigand while we could switch our definitions to IntPtr, I believe there must be a way to create simple wrapper types (which *Reference types are) on any platform that would be compatible with ABI.

Would this be solvable with MarshalAs?

@uweigand
Copy link

uweigand commented Jul 7, 2023

Unfortunately I don't think MarshalAs works here, since this is not supported with delegate* function pointers. Trying to add a MarshalAs attribute results in a syntax error, and indeed the language spec says:

Custom attributes cannot be applied to a delegate* or any of its elements.

@filmor
Copy link
Member

filmor commented Jul 7, 2023

That's very unfortunate... @lostmsu I don't really see a way to fix this apart from using IntPtr for the delegates again and doing the "conversion" to New/BorrowedReference in a wrapper.

@Sapana-Khemkar
Copy link

@filmor @lostmsu
Can you consider fixing this issue in upcoming release? This will enable pythonnet on both s390 and ppc64le platform.

@lostmsu
Copy link
Member

lostmsu commented Jul 13, 2023

@Sapana-Khemkar currently there's no upcoming release planned.

@kkle1994
Copy link

When I was trying to load protobuf-net, it failed all the times. I guess there is some similar issue as this.

@lostmsu
Copy link
Member

lostmsu commented Sep 11, 2024

Me and @filmor can not support ppc64le and s390 because we have no devices with those architectures, and nowhere to conveniently host an actions runner for these platforms.

If someone with access to such devices would fix the issue, send a PR, and it won't break x64 and ARM64, we can accept it. But we won't be developing such a fix ourselves.

@lostmsu lostmsu changed the title Delegates declared using NewReference and BorrowedReference structs are not working/crashing on ppc64le/s390x architectures ppc64le/s390x support Sep 11, 2024
@lostmsu lostmsu changed the title ppc64le/s390x support Delegates declared using NewReference and BorrowedReference structs are not working/crashing on ppc64le/s390x Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
not planned - PRs welcome
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@filmor @lostmsu @uweigand @kkle1994 @Sapana-Khemkar @alhad-deshpande