Unimplemented or broken methods in Python wrappers to standard C# containers #2531
Description
Environment
- Pythonnet version: 3.0.5
- Python version: 3.13
- Operating System: MacOS Sequoia 15.1.1
- .NET Runtime: 8.0.403
Details
After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (Sequence, MutableSequence, Mapping, MutableMapping, ...).
Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.
Shared setup
import pythonnet
pythonnet.load('coreclr')
import clr
from System.Collections.Generic import Dictionary, List, KeyValuePair
from System.Collections.Immutable import ImmutableArray, ImmutableList, ImmutableDictionary
from System import Int32, String, Object, Nullable, Array
from System.Collections.Generic import CollectionExtensionsArray-like containers
ArrT = Array[Int32]
ImmArrT = ImmutableArray[Int32]
ListT = List[Int32]
arr = ArrT([0, 1, 2, 3, 4])
immarr = ImmutableArray.Create[Int32](0, 1, 2, 3, 4) # is this supposed to work, instead of using Create()?
lst = ListT(ArrT([0, 1, 2, 3, 4]))
immlst = ImmutableList.ToImmutableList[Int32](ArrT([0, 1, 2, 3, 4]))What worked everywhere
__iter____reversed____contains____len__reverse()__getitem__when the index is within boundsindex()when the element is in the container__setitem__when the index is within bounds
What didn't work
__getitem__when the index is negative or out of bounds
arr[-1] # OK
arr[999] # OK, raises IndexError
immarr[-1] # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst[-1] # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst[-1] # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')Python expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of pop() from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.
Also, the exception when the index is out of bounds should be IndexError, otherwise this breaks the implementation of Sequence.index() among other things.
Possible fix
Add the following method to class SequenceMixin:
def __getitem__(self, key):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
return self.get_Item(key)... and make sure List doesn't override it?
index()when the element is NOT in the container
arr.index(999) # OK, raises ValueError
immarr.index(999) # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst.index(999) # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst.index(999) # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')This is caused by the previous point, the exceptions are thrown in Sequence.index().
__getitem__,__setitem__and__delitem__with slices
arr[0:3] # TypeError: array index has type slice, expected an integer
immarr[0:3] # TypeError: No method matches given arguments for ImmutableArray`1.get_Item: (<class 'slice'>)
lst[0:3] # TypeError: No method matches given arguments for List`1.get_Item: (<class 'slice'>)
immlst[0:3] # TypeError: No method matches given arguments for ImmutableList`1.get_Item: (<class 'slice'>)This is ok, clearly not supported which is fine. Exceptions self-explanatory.
__setitem__when the index is negative or out of bounds
arr[-1] = 9 # OK
lst[-1] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
arr[999] = 9 # OK, raises IndexError
lst[999] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')OK for Array; for List, same as __getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.
Possible fix
Like for __getitem__, add the following method to class MutableSequenceMixin:
def __setitem__(self, key, value):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
self.set_Item(key, value)- methods that would result in a different
Arraylength
del arr[4] # SystemError: Objects/longobject.c:583: bad argument to internal function
arr.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
arr.append(5) # IndexError; uses abc implementation, which falls back to insert
arr.pop() # SystemError (same as del); uses abc implementation, which calls del
arr.clear() # SystemError (same as del); uses abc implementation, which calls del
arr.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
arr.remove(9) # SystemError (same as del); uses abc implementation, which calls del
arr += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extendThis could be fine, to disallow changing an array's length, even though Array.Resize and Array.Copy could in theory allow us to have all of these working and be fairly efficient.
But the exceptions have me think that this is another kind of problem:
- the
SystemErrorcomes from the bowels of CPython, precisely here, so I assume this is unintended behavior; - the
IndexErroris caused byMutableSequenceMixinnot implementinginsert(); if not supporting this is intended, it could be made more understandable by implementing it (inArray, notMutableSequenceMixin) and throwingTypeError,System.NotSupportedExceptionor something more explanatory.
- deleting
Listelements
del lst[0] # crashCrash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 37819 abort python3 delme_test_pythonnet.py
Like for Dictionary, __delitem__ on a List causes a hard crash; possibly related to #2530.
- other methods mutating
Listlength
lst.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
lst.append(5) # IndexError; uses abc implementation, which calls insert
lst.pop() # IndexError; uses abc implementation, which calls __getitem__(-1) and del
lst.clear() # IndexError; uses abc implementation, which calls pop
lst.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
lst.remove(3) # CRASH; uses abc implementation, which calls del
lst += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extendThese should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and insert() not being implemented in MutableMappingMixin.
Possible fix
Add the following method to class MutableSequenceMixin:
def insert(self, index, value):
self.Insert(index, value)And also apply some of the previous proposed fixes, on top of fixing del.
Dictionary-like containers
DictT = Dictionary[Int32, String]
DictT2 = Dictionary[String, Int32]
DictT3 = Dictionary[String, Nullable[Int32]]
d = DictT()
d[10] = "10"
d[20] = "20"
d[30] = "30"
d2 = DictT2()
d3 = DictT3()
rod = CollectionExtensions.AsReadOnly[Int32, String](DictT(d)) # ReadOnlyDictionary<int, string>
immd = ImmutableDictionary.ToImmutableDictionary[Int32, String](d) # ImmutableDictionary<int, string>What worked everywhere
__contains____len____getitem__when the key is in the dictionary__setitem__keysvaluesitems, although maybe it could be achieved without copyinggetclearupdate
What didn't work
- Deleting elements
del d[""] # CRASHThis crashes with the following output; already tracked in #2530.
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 44289 abort python3 delme_test_pythonnet.py
__iter__returnsKeyValuePairs instead of tuples
list(d) # gives a list of KeyValuePairThis contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the popitem() implementation from collections.abc. At least the latter should be fixed.
popitem
d.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[Int32,String]'>)
d2.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Int32]'>)
d3.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Nullable[Int32]]'>)This is caused by the previous point, the exceptions are thrown in MutableMapping.popitem() which expects the dictionary to be an iterable of its keys.
Possible fix
Add the following to MutableMappingMixin:
def popitem(self):
try:
key = next(iter(self.Keys))
except StopIteration:
raise KeyError from None
value = self[key]
del self[key]
return key, value... on top of fixing del.
__getitem__when the key is not in the dictionary
d[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
rod[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
immd[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.The exception is not translated to a ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.
popandsetdefaultfor some types
d.pop(10) # OK
d2.pop("10") # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.pop("10") # OK
d.setdefault(10, "10") # OK
d2.setdefault("10", 10) # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.setdefault("10", 10) # OK
d3.setdefault("10") # Also OKI believe this breaks only with non-nullable value types: the exception is called from the MutableMappingMixin implementation of pop(), which calls self.TryGetValue(key, None) here.
Possible fix: call self.TryGetValue(key) instead? I believe this is supported since commit f69753c.