Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Unimplemented or broken methods in Python wrappers to standard C# containers #2531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DiegoBaldassarMilleuno opened this issue Dec 17, 2024 · 3 comments

Comments

@DiegoBaldassarMilleuno
Copy link

Environment

  • Pythonnet version: 3.0.5
  • Python version: 3.13
  • Operating System: MacOS Sequoia 15.1.1
  • .NET Runtime: 8.0.403

Details

After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (Sequence, MutableSequence, Mapping, MutableMapping, ...).
Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.

Shared setup

import pythonnet
pythonnet.load('coreclr')
import clr
from System.Collections.Generic import Dictionary, List, KeyValuePair
from System.Collections.Immutable import ImmutableArray, ImmutableList, ImmutableDictionary
from System import Int32, String, Object, Nullable, Array
from System.Collections.Generic import CollectionExtensions

Array-like containers

ArrT = Array[Int32]
ImmArrT = ImmutableArray[Int32]
ListT = List[Int32]

arr = ArrT([0, 1, 2, 3, 4])
immarr = ImmutableArray.Create[Int32](0, 1, 2, 3, 4)  # is this supposed to work, instead of using Create()?
lst = ListT(ArrT([0, 1, 2, 3, 4]))
immlst = ImmutableList.ToImmutableList[Int32](ArrT([0, 1, 2, 3, 4]))

What worked everywhere

  • __iter__
  • __reversed__
  • __contains__
  • __len__
  • reverse()
  • __getitem__ when the index is within bounds
  • index() when the element is in the container
  • __setitem__ when the index is within bounds

What didn't work

  • __getitem__ when the index is negative or out of bounds
arr[-1]     # OK
arr[999]    # OK, raises IndexError
immarr[-1]  # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst[-1]     # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst[-1]  # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')

Python expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of pop() from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.
Also, the exception when the index is out of bounds should be IndexError, otherwise this breaks the implementation of Sequence.index() among other things.

Possible fix

Add the following method to class SequenceMixin:

    def __getitem__(self, key):
        length = len(self)
        key = key if key >= 0 else length + key
        if key not in range(length):
            raise IndexError('index out of range')
        return self.get_Item(key)

... and make sure List doesn't override it?

  • index() when the element is NOT in the container
arr.index(999)     # OK, raises ValueError
immarr.index(999)  # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst.index(999)     # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst.index(999)  # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')

This is caused by the previous point, the exceptions are thrown in Sequence.index().

  • __getitem__, __setitem__ and __delitem__ with slices
arr[0:3]     # TypeError: array index has type slice, expected an integer
immarr[0:3]  # TypeError: No method matches given arguments for ImmutableArray`1.get_Item: (<class 'slice'>)
lst[0:3]     # TypeError: No method matches given arguments for List`1.get_Item: (<class 'slice'>)
immlst[0:3]  # TypeError: No method matches given arguments for ImmutableList`1.get_Item: (<class 'slice'>)

This is ok, clearly not supported which is fine. Exceptions self-explanatory.

  • __setitem__ when the index is negative or out of bounds
arr[-1] = 9   # OK
lst[-1] = 9   # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
arr[999] = 9  # OK, raises IndexError
lst[999] = 9  # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

OK for Array; for List, same as __getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.

Possible fix

Like for __getitem__, add the following method to class MutableSequenceMixin:

    def __setitem__(self, key, value):
        length = len(self)
        key = key if key >= 0 else length + key
        if key not in range(length):
            raise IndexError('index out of range')
       self.set_Item(key, value)
  • methods that would result in a different Array length
del arr[4]                # SystemError: Objects/longobject.c:583: bad argument to internal function
arr.insert(0, 5)          # IndexError; calls abc implementation, which is a stub!
arr.append(5)             # IndexError; uses abc implementation, which falls back to insert
arr.pop()                 # SystemError (same as del); uses abc implementation, which calls del
arr.clear()               # SystemError (same as del); uses abc implementation, which calls del
arr.extend([10, 11, 12])  # IndexError; uses abc implementation, which calls append
arr.remove(9)             # SystemError (same as del); uses abc implementation, which calls del
arr += [10, 11, 12]       # IndexError; uses abc implementation, which falls back to extend

This could be fine, to disallow changing an array's length, even though Array.Resize and Array.Copy could in theory allow us to have all of these working and be fairly efficient.
But the exceptions have me think that this is another kind of problem:

  1. the SystemError comes from the bowels of CPython, precisely here, so I assume this is unintended behavior;
  2. the IndexError is caused by MutableSequenceMixin not implementing insert(); if not supporting this is intended, it could be made more understandable by implementing it (in Array, not MutableSequenceMixin) and throwing TypeError, System.NotSupportedException or something more explanatory.
  • deleting List elements
del lst[0]  # crash
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
   at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
   at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
   at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1]    37819 abort      python3 delme_test_pythonnet.py

Like for Dictionary, __delitem__ on a List causes a hard crash; possibly related to #2530.

  • other methods mutating List length
lst.insert(0, 5)          # IndexError; calls abc implementation, which is a stub!
lst.append(5)             # IndexError; uses abc implementation, which calls insert
lst.pop()                 # IndexError; uses abc implementation, which calls __getitem__(-1) and del
lst.clear()               # IndexError; uses abc implementation, which calls pop
lst.extend([10, 11, 12])  # IndexError; uses abc implementation, which calls append
lst.remove(3)             # CRASH; uses abc implementation, which calls del
lst += [10, 11, 12]       # IndexError; uses abc implementation, which falls back to extend

These should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and insert() not being implemented in MutableMappingMixin.

Possible fix

Add the following method to class MutableSequenceMixin:

    def insert(self, index, value):
        self.Insert(index, value)

And also apply some of the previous proposed fixes, on top of fixing del.

Dictionary-like containers

DictT  = Dictionary[Int32, String]
DictT2 = Dictionary[String, Int32]
DictT3 = Dictionary[String, Nullable[Int32]]

d = DictT()
d[10] = "10"
d[20] = "20"
d[30] = "30"
d2 = DictT2()
d3 = DictT3()
rod = CollectionExtensions.AsReadOnly[Int32, String](DictT(d))  # ReadOnlyDictionary<int, string>
immd = ImmutableDictionary.ToImmutableDictionary[Int32, String](d)  # ImmutableDictionary<int, string>

What worked everywhere

  • __contains__
  • __len__
  • __getitem__ when the key is in the dictionary
  • __setitem__
  • keys
  • values
  • items, although maybe it could be achieved without copying
  • get
  • clear
  • update

What didn't work

  • Deleting elements
del d[""]  # CRASH

This crashes with the following output; already tracked in #2530.

Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
   at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
   at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
   at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1]    44289 abort      python3 delme_test_pythonnet.py
  • __iter__ returns KeyValuePairs instead of tuples
list(d)  # gives a list of KeyValuePair

This contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the popitem() implementation from collections.abc. At least the latter should be fixed.

  • popitem
d.popitem()   # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[Int32,String]'>)
d2.popitem()  # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Int32]'>)
d3.popitem()  # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Nullable[Int32]]'>)

This is caused by the previous point, the exceptions are thrown in MutableMapping.popitem() which expects the dictionary to be an iterable of its keys.

Possible fix

Add the following to MutableMappingMixin:

   def popitem(self):
       try:
           key = next(iter(self.Keys))
       except StopIteration:
           raise KeyError from None
       value = self[key]
       del self[key]
       return key, value

... on top of fixing del.

  • __getitem__ when the key is not in the dictionary
d[40]     # KeyNotFoundException: The given key '40' was not present in the dictionary.
rod[40]   # KeyNotFoundException: The given key '40' was not present in the dictionary.
immd[40]  # KeyNotFoundException: The given key '40' was not present in the dictionary.

The exception is not translated to a ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.

  • pop and setdefault for some types
d.pop(10)   # OK
d2.pop("10")  # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.pop("10")  # OK
d.setdefault(10, "10")   # OK
d2.setdefault("10", 10)  # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.setdefault("10", 10)  # OK
d3.setdefault("10")      # Also OK

I believe this breaks only with non-nullable value types: the exception is called from the MutableMappingMixin implementation of pop(), which calls self.TryGetValue(key, None) here.
Possible fix: call self.TryGetValue(key) instead? I believe this is supported since commit f69753c.

@filmor
Copy link
Member

filmor commented Dec 17, 2024

Thank you for the systematic write-up! It will take me some time to go through all of these, but it would be great if you could create a draft PR with the tests that you ran for this analysis. That will make it easier to iterate on the fixes.

@lostmsu
Copy link
Member

lostmsu commented Dec 18, 2024

We can not really implement negative indexing in general case because neither Python requires it nor .NET guarantees that negative index values can not be used for other purposes (e.g. it would be a breaking change).

DiegoBaldassarMilleuno added a commit to DiegoBaldassarMilleuno/pythonnet that referenced this issue Dec 20, 2024
Adding testing for adherence to collections.abc protocols for the
following classes: Array, List, ImmutableArray, ImmutableList,
Dictionary, ImmutableDictionary and ReadOnlyDictionary
Tests for Python list and dict are also present as a
reference but commented out.
DiegoBaldassarMilleuno added a commit to DiegoBaldassarMilleuno/pythonnet that referenced this issue Dec 20, 2024
Adding testing for adherence to collections.abc protocols for the
following classes: Array, List, ImmutableArray, ImmutableList,
Dictionary, ImmutableDictionary and ReadOnlyDictionary
Tests for Python list and dict are also present as a
reference but commented out.
DiegoBaldassarMilleuno added a commit to DiegoBaldassarMilleuno/pythonnet that referenced this issue Dec 20, 2024
Adding testing for adherence to collections.abc protocols for the
following classes: Array, List, ImmutableArray, ImmutableList,
Dictionary, ImmutableDictionary and ReadOnlyDictionary
Tests for Python list and dict are also present as a
reference but commented out.
@DiegoBaldassarMilleuno
Copy link
Author

Sorry for the delay, been busy.
Submitted draft PR #2535 as requested

We can not really implement negative indexing in general case because neither Python requires it nor .NET guarantees that negative index values can not be used for other purposes (e.g. it would be a breaking change).

Pity, well we can still fix pop(); even if the negative index thing isn't required explicitly, the implementation in the Python Standard Library implicitly relies on it.

Another thing, i found out that items() is broken in Dictionary[K,V] when V is a value type, for the same reason as pop and setdefault.

I'm going on holiday for a couple of weeks, so I won't be available for a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants