Operator overloads support #1324

christabella · 2020-12-16T23:28:35Z

What does this implement/fix? Explain your changes.

Adapts @amos402's PR #907 for the latest Pythonnet version. I can also modify Amos's PR directly if that is more convenient for @amos402 @lostmsu who have already made comments on it.

Tests are kept the same, as well as the links in classmanager, typemanager, and runtime.
However, the methodbinder logic in Amos's PR is quite different from the current methodbinder and cannot be copied directly.
- Renamed 2 variables pynargs and clrnargs for readability and consistency with the rest of methodbinder.
Added specialized slot types (e.g. nb_subtract, nb_and) from the PyNumberMethods struct. Because a recent PR https://github.com/pythonnet/pythonnet/pull/1292/files defined the TypeOffset and ITypeOffset classes separately such that they no longer depends on interop3x.cs of a specific Python version.
Put bit shift operators (<< and >>) a separate test because they are quite different from the rest of the operators (+/-/& etc.)---the C# language specification is such that that shift operators always need an int for the second arg: https://stackoverflow.com/questions/7586887/c-sharp-shift-operator-overload. Instead of doing c = a >> b, we have to do c = a >> b.Num or get an ArgumentException.
Added other operators including reverse operators (e.g. __radd__).

Although Amos's PR had some params array tests, it looks like params array handling is currently being looked into by @filmor (#1304 and #1311), so I removed those for now, since they are not related to operator overloading.

Does this close any currently open issues?

Fixes #906.

Suggestions/future work

Unary operators

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
If an enhancement PR, please create docs and at best an example
Can add docs once reverse operators are handled.
Add yourself to AUTHORS
Updated the CHANGELOG

christabella · 2020-12-17T09:45:25Z

2 failing tests for Python 3.9 on windows, but it seems to be an unrelated test test_subclass.py:test_events().

I tried reproduce this locally on my windows laptop using python 3.9.0 but got errors such as
System.NotSupportedException : Python ABI v3.9.0 is not supported

src/runtime/operatormethod.cs

tminka · 2020-12-17T13:53:51Z

src/runtime/typemanager.cs

@@ -280,6 +280,7 @@ internal static IntPtr CreateType(ManagedType impl, Type clrType)
            {
                throw new PythonException();
            }
+            OperatorMethod.FixupSlots(type, clrType);


Please address the comment that was made on this file in PR #907

I have moved it up before pytype_ready

lostmsu

Overall looks good as a starting point. We need to keep in mind, that his does not give us full operator support.

In C# and .NET only one argument of a binary operator must match the type, that defines it, and it does not have to be the first argument. This needs to be fixed in this PR: when adding operator methods only the methods where the left argument is of the parent type must be considered (and maybe also inherited ones). I think without this attempts to call the following overloaded operator on two instances of OperableObject would fail from Python:

class OperableObject {
  ... OperableObject operator+(int left, OperableObject right) => ...;
  ... OperableObject operator+(OperableObject left, OperableObject right) => ...;
}

Binary operators, that have the first argument of different type need to be handled correctly. I think this should be done in a separate PR, because I expect the amount of work to be comparable or even greater: loading a new type might add overloads to existing type's __add__ for example.
A few operators are not covered here. The full list in C# is: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/operator-overloading . Notably missing comparison, bitwise, and obscure stuff like conversion to true/false. I suggest we add them in a separate PR.

src/runtime/methodbinder.cs

src/runtime/operatormethod.cs

src/embed_tests/TestPyMethod.cs

src/runtime/classderived.cs

src/runtime/operatormethod.cs

christabella · 2020-12-20T14:13:23Z

Thank you for the review. Handling operators with asymmetrical types is a really good point. I have added the "forward" case where the arguments are OperableObject and int; I will add the "reverse" case where the arguments are int and OperableObject.

lostmsu · 2020-12-21T04:17:31Z

@christabella I think the reverse case can be done in a separate PR. Until it is in, do you want me to review this one?

christabella · 2020-12-21T08:25:34Z

@lostmsu Yes please, would appreciate your review once again.

amos402

The original PR #907 reduced some iterations, redundant checking, and heap allocation in order to improve the performance for the binding behavior, also it included the new style params matching, so it was quite different from the master branch.
If you don't need those enhance you don't need to port them all.

src/runtime/methodbinder.cs

src/embed_tests/TestOperator.cs

src/runtime/methodbinder.cs

lostmsu

@christabella please, revert rename to pynargs too.

src/embed_tests/TestOperator.cs

src/runtime/operatormethod.cs

src/runtime/methodbinder.cs

src/runtime/operatormethod.cs

…ads.

Also move fixupslots before pytype_ready. Also apply access violation fix in classderived.

lostmsu · 2020-12-28T22:42:57Z

@christabella this looks good to me. But considering the upcoming forward operator support and the OperatorBinder are going to rewrite parts of this PR, it will make sense to just continue updating this PR.

christabella · 2020-12-29T02:17:03Z

@christabella this looks good to me. But considering the upcoming forward operator support and the OperatorBinder are going to rewrite parts of this PR, it will make sense to just continue updating this PR.

@lostmsu sure I can do that, if you think it makes sense having seen the upcoming christabella#1

lostmsu · 2020-12-29T02:34:47Z

@christabella actually, I suggested to create an OperatorBinder class and possibly OperatorBinding class similar to existing ConstructorBinder and ConstructorBinding, which also do special argument handling, but for constructors.

You could probably also move some/all of operatormethod.cs into it.

But! It is not a requirement, you can also just leave code as-is.

christabella · 2020-12-29T13:48:40Z

@christabella actually, I suggested to create an OperatorBinder class and possibly OperatorBinding class similar to existing ConstructorBinder and ConstructorBinding, which also do special argument handling, but for constructors.

You could probably also move some/all of operatormethod.cs into it.

But! It is not a requirement, you can also just leave code as-is.

@lostmsu thanks for clarifying, I understand now. So that would be quite a refactor, for now I would try to keep this PR as-is.

src/runtime/methodbinder.cs

tminka · 2020-12-29T17:48:54Z

src/runtime/classmanager.cs

+                    // Only methods where the left operand is the declaring type.
+                    ci.members[pyName] = new MethodObject(type, name, forwardMethods);
+                    // Only methods where only the right operand is the declaring type.
+                    ci.members[pyNameReverse] = new MethodObject(type, name, reverseMethods);


What if there are no reverseMethods?

Currently it will just create a method that's never used; I can fix that by adding a hasReverse field in the SlotDefinition struct.

So if C# reverse operator isn't defined, so the Python reverse name will have an empty method list in MethodObject

So if a user calls b.__radd__(a) by writing a + b where there is no forward +() defined on a so Python resorts to __radd__, AND no reverse +() defined on b, what happens? Wouldn't hurt to add a check for an empty reverseMethods before doing ci.members[pyNameReverse] = new MethodObject(type, name, reverseMethods);

Another unrelated issue is that of doing unnecessary work, if the Python slot doesn't support reverse e.g. unary operators (__rtruediv__ exists even though python docs don't list them)

It is also possible to have no "forward" methods.

If no reverse methods are defined, the corresponding slot should not be set. Same for forward.

Python slot doesn't support reverse e.g. __truediv__

Not sure what you mean. There's __rtruediv__

That's true, I've just put a check then to not set the slot method if the method list is empty

src/runtime/classmanager.cs

src/runtime/methodbinder.cs

src/embed_tests/TestOperator.cs

tminka · 2020-12-29T18:14:16Z

src/embed_tests/TestOperator.cs

+cls = {name}
+a = cls(2)
+b = cls(10)
+c = a + b


At this point, both of these calls will succeed (and do the same thing):

a.op_Addition(b) a.op_Addition(a,b)

Is that desirable behavior?

So the first statement would have previously failed and now succeeds. I guess this is a question for @lostmsu if we want to let something that's wrongly written, not fail

a.op_Addition(a, b) must be failing for operators, defined in C#.

The code I wrote is Python code. If a.op_Addition(a,b) now fails in Python.NET, then it will break existing code using Python.NET.

Breaking should not be a big deal, since we are making major version bump with (hopefully) a good overhaul.
But I don't think a.op_Addition(a, b) should be working now.

src/runtime/methodbinder.cs

src/embed_tests/TestOperator.cs

This partially reverts commit 8cce61d. Due to breaking change

src/runtime/methodbinder.cs

src/runtime/operatormethod.cs

CHANGELOG.md

src/runtime/classmanager.cs

christabella changed the title ~~[WIP] Operator overloads support~~ Operator overloads support Dec 17, 2020

tminka reviewed Dec 17, 2020

View reviewed changes

src/runtime/operatormethod.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 17, 2020

View reviewed changes

christabella force-pushed the feat/operator-overloads branch from e0ebf2c to b2053a7 Compare December 18, 2020 14:36

lostmsu requested changes Dec 18, 2020

View reviewed changes

christabella force-pushed the feat/operator-overloads branch 3 times, most recently from c0b5569 to 659ad56 Compare December 20, 2020 14:10

amos402 reviewed Dec 21, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

christabella force-pushed the feat/operator-overloads branch 2 times, most recently from 2dd951d to 90cd7d3 Compare December 21, 2020 11:13

christabella requested a review from lostmsu December 21, 2020 11:43

lostmsu requested changes Dec 21, 2020

View reviewed changes

src/embed_tests/TestOperator.cs Show resolved Hide resolved

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

christabella force-pushed the feat/operator-overloads branch from 90cd7d3 to 3438982 Compare December 21, 2020 19:32

christabella requested a review from lostmsu December 22, 2020 10:39

lostmsu requested changes Dec 22, 2020

View reviewed changes

tminka reviewed Dec 22, 2020

View reviewed changes

src/embed_tests/TestOperator.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 22, 2020

View reviewed changes

src/runtime/operatormethod.cs Show resolved Hide resolved

tminka reviewed Dec 22, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 22, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 22, 2020

View reviewed changes

src/runtime/operatormethod.cs Show resolved Hide resolved

christabella added 5 commits December 23, 2020 09:15

Overload operators except for bit shifts and default args.

5130aaa

Remove params array tests since they are unrelated to operator overlo…

0d0ca87

…ads.

Fix type bug; rename variables pynargs etc.

253f9c7

Remove unused variables and add comments.

8cdb61c

Add comments and remove unused in operatormethod.cs

c26e589

Also move fixupslots before pytype_ready. Also apply access violation fix in classderived.

christabella requested a review from lostmsu December 28, 2020 08:01

Support reverse binary operations

e7da0bc

Use reverse instead of forward (semantics)

a376838

christabella force-pushed the feat/operator-overloads branch from 2fe1de7 to a376838 Compare December 29, 2020 14:44

tminka reviewed Dec 29, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 29, 2020

View reviewed changes

src/runtime/classmanager.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 29, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

tminka reviewed Dec 29, 2020

View reviewed changes

src/embed_tests/TestOperator.cs Show resolved Hide resolved

tminka reviewed Dec 29, 2020

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

christabella added 2 commits December 30, 2020 14:43

Address @tminka's comments

6923a78

Support unary neg and pos operators

4c992d8

tminka reviewed Dec 30, 2020

View reviewed changes

src/embed_tests/TestOperator.cs Show resolved Hide resolved

christabella added 2 commits January 4, 2021 10:40

Remove isOperator from MatchesArgumentCount (simplify), add test.

8cce61d

Revert "Remove isOperator from MatchesArgumentCount (simplify)"

09a2047

This partially reverts commit 8cce61d. Due to breaking change

tminka reviewed Jan 4, 2021

View reviewed changes

src/runtime/methodbinder.cs Outdated Show resolved Hide resolved

christabella added 2 commits January 4, 2021 16:34

Properly remove isOperator from MatchesArgumentCount (@tminka comment)

41bd07f

Update changelog.

10ccf1e

tminka reviewed Jan 4, 2021

View reviewed changes

src/runtime/operatormethod.cs Outdated Show resolved Hide resolved

Add ones complement and modulo tests (@tminka comment)

5682e0c

lostmsu approved these changes Jan 5, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

src/runtime/classmanager.cs Show resolved Hide resolved

Merge branch 'master' into feat/operator-overloads

5f45c70

lostmsu merged commit 9f01ebb into pythonnet:master Jan 5, 2021

This was referenced Jan 6, 2021

Support comparison operators #1347

Merged

Basic operator overloads support #907

Closed

Operator overloads support #1324

Operator overloads support #1324

Uh oh!

Conversation

christabella commented Dec 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Suggestions/future work

Checklist

Uh oh!

christabella commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lostmsu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christabella commented Dec 20, 2020

Uh oh!

lostmsu commented Dec 21, 2020

Uh oh!

christabella commented Dec 21, 2020

Uh oh!

amos402 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lostmsu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lostmsu commented Dec 28, 2020

Uh oh!

christabella commented Dec 29, 2020

Uh oh!

lostmsu commented Dec 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christabella commented Dec 29, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christabella Dec 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

christabella commented Dec 16, 2020 •

edited

Loading

christabella commented Dec 17, 2020 •

edited

Loading

lostmsu left a comment •

edited

Loading

lostmsu commented Dec 29, 2020 •

edited

Loading

christabella Dec 30, 2020 •

edited

Loading

tminka Dec 30, 2020 •

edited

Loading