Proposal: Safe pointers #1043

lostmsu · 2020-02-13T23:43:44Z

What does this implement/fix? Explain your changes.

With C# 7.3 ref structs and C# 8.0 using working on any method named Dispose it is possible to track new vs borrowed references at compile time.

This proposal introduces BorrowedReference and NewReference types, that are ref-only structs, meaning you can't put them into PyObject directly, so they can implement different ToPyObject methods, one doing IncRef, and the other one not (alternatively, we can have PyObject constructor overloads). Along with Roslyn-based NonCopyableAnalyzer and FxCop after dotnet/roslyn-analyzers#3305 is fixed, this can ensure, that we do reference counting correctly at compile time.

We can gradually change return and parameter types in PInvoke methods in Runtime to one of BorrowedReference or NewReference to take advantage of this.

Downsides

~~Build system must support C# 8.0, which is currently not supported in all build configurations.~~
Without C# 8.0 one would have to dispose NewReference instances with try ... finally ... block.

lostmsu · 2020-02-13T23:45:47Z

src/runtime/methodbinder.cs

@@ -292,7 +292,7 @@ internal Binding Bind(IntPtr inst, IntPtr args, IntPtr kw, MethodBase info, Meth
                for (int i = 0; i < pynkwargs; ++i)
                {
                    var keyStr = Runtime.GetManagedString(Runtime.PyList_GetItem(keylist, i));
-                    kwargDict[keyStr] = Runtime.PyList_GetItem(valueList, i);
+                    kwargDict[keyStr] = Runtime.PyList_GetItem(valueList, i).DangerousGetAddress();


This generates a compiler warning, that uncovers a potentially dangerous pattern, where kwargsDict may contain borrowed references.

lostmsu · 2020-02-13T23:46:38Z

src/runtime/pydict.cs

@@ -139,12 +139,13 @@ public PyObject Values()
        /// </remarks>
        public PyObject Items()
        {
-            IntPtr items = Runtime.PyDict_Items(obj);
-            if (items == IntPtr.Zero)
+            using var items = Runtime.PyDict_Items(this.obj);


After dotnet/roslyn-analyzers#3305 is fixed we can configure build to fail here if using would be omitted

lostmsu · 2020-02-14T00:06:18Z

@amos402 can you check this one out too?

codecov-io · 2020-02-14T03:54:56Z

Codecov Report

Merging #1043 into master will increase coverage by 3.64%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1043      +/-   ##
==========================================
+ Coverage   83.11%   86.75%   +3.64%     
==========================================
  Files           1        1              
  Lines         302      302              
==========================================
+ Hits          251      262      +11     
+ Misses         51       40      -11

Flag	Coverage Δ
#setup_linux	`65.56% <ø> (ø)`	⬆️
#setup_windows	`71.52% <ø> (+6.62%)`	⬆️

Impacted Files	Coverage Δ
setup.py	`86.75% <0%> (+3.64%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5548e3...6ae63cd. Read the comment docs.

amos402 · 2020-02-14T04:39:13Z

You can also implement an implicit operator for PyObject make it easy for usage.
Reference struct also can provide more interfaces, like RefCnt, ToString, ect.
By the by, Py.GILState can be strcuct easily too.

lostmsu · 2020-02-22T21:24:07Z

@filmor I'd love if we get this in and take as new standard for coding Python handle manipulations, as it would make otherwise hard-to-debug segmentation fault/access violation bugs easier to prevent/spot/debug.

lostmsu · 2020-02-22T21:31:39Z

@amos402 I can proceed to merge it, if you can sign-off a review, and are for this change in general.

I do not plan to extend the scope yet (like making PyObject changes to take these, etc). We can do it in the future PRs.

amos402 · 2020-02-23T11:45:45Z

src/runtime/Python.Runtime.csproj

@@ -83,6 +83,7 @@
    </Compile>
    <Compile Include="arrayobject.cs" />
    <Compile Include="assemblymanager.cs" />
+    <Compile Include="BorrowedReference.cs" />


Is it good for separate these three items to different files? They're small, related, and it make the project files be scattered.

The might get more code later on. Generally, I prefer a type per file approach for anything except delegate types.

Or create a folder for them? Just just too far if they sorted by name I thought🤣

amos402 · 2020-02-23T11:54:50Z

src/runtime/NewReference.cs

+{
+    using System;
+    [NonCopyable]
+    ref struct NewReference


Is it necessary to limit it as a stack-allocated value? If make it to a normal struct and rename it to PyReference, it can be assigned to a collection and implement the IDisposable for using () usage.
For better usage, a BorrowedReference may be able to promote to a PyReference for using the implict operate overloading.

With [NoCopyable] you won't be able to put it into a collection anyway. And without [NoCopyable] it does not provide any safety guarantees.

Having it as ref struct saves you from adding ref everywhere you take it.

But....sometimes I just want to put them into collections...😂

Then you can still do DangerousGetHandle or create a PyObject from them.

But this type also has a purpose for reminding others that it's a reference don't forget to decref it, isn't it?

Because it must be clear from code, that care is necessary around this scenario.

Also, as mentioned above, [NoCopyable] would already prevent you from putting an instance into a collection, even if ref were removed. And [NoCopyable] is basically the sole purpose of this change, as if you do

NewReference refA = GetNewReferenceSomehow(); NewReference refB = refA;

Then you already miappropriated refcounts.

Maybe more introduce a type named PyHandle for that?

emmm, but that make BorrowedReference embarrassed.

@amos402 The idea with PyHandle can be reviewed separately. This PR is only for tracking new vs borrowed references as used in Python documentation for C API.

amos402 · 2020-02-23T11:57:31Z

src/runtime/NewReference.cs

+        public IntPtr Pointer { get; set; }
+        public bool IsNull => this.Pointer == IntPtr.Zero;
+
+        public PyObject ToPyObject()


I prefer two methods:
ToPyObject: inref and create a PyObject, not assign the Pointer to nullptr
AsPyObject: use this implementation

Constructing a new PyObject from references should be done using PyObject constructors (which can be added later). I agree though, that ToPyObject is not descriptive enough. Maybe MoveToPyObject or IntoPyObject?

amos402 · 2020-02-23T12:16:42Z

src/runtime/BorrowedReference.cs

+        }
+
+        [Obsolete("Use overloads, that take BorrowedReference or NewReference")]
+        public IntPtr DangerousGetAddress()


Use a property named Address can be more fit?
Why I have to be warned if I trying use the raw pointer, seems it doesn't make any sense. Or just don't expose the interface and use explicit operator IntPtr instead.

This is modeled after SafeHandle class and its DangerousGetHandle method.
The whole point of introducing these references is to get rid of IntPtrs entirely. It should warn you against converting into IntPtr. Maybe Obsolete part is unnecessary as Dangerous sounds like a warning enough. I will remove Obsolete.

removed public property Pointer from NewReference and replaced with DangerousGetAddress

lostmsu · 2020-02-26T02:01:17Z

@amos402 @filmor I urge you to review this ASAP. As I am investigating segfaults in #1050 and #1060 , I'd love to have type checker on my side. Will definitely bring this up during the next call.

BTW, I don't think it is necessary to update branch with the latest master every time before merging, as long as there are no conflicts. In an unlikely case, when it breaks the build, the change can be reverted and the PR reopened.

filmor · 2020-02-26T11:20:41Z

I like this a lot, I'm was just a bit reluctant about using C# 8 as it won't be officially supported on .NET Framework, but apparently you are not doing that yet.

* NewReference type and an example usage * BorrowedReference + example, that exposes dangerous pattern * make BorrowedReference readonly ref struct * BorrowedReference.Pointer is a private readonly field * renamed NewReference.ToPyObject to MoveToPyObject * removed public property Pointer from NewReference and replaced with DangerousGetAddress

lostmsu added 5 commits February 13, 2020 14:42

NewReference type and an example usage

4a84a1e

BorrowedReference + example, that exposes dangerous pattern

88944a2

actually, borrowed refrences can be copied

e1ba92c

make BorrowedReference readonly ref struct

2d11f82

updated old project file

dbb3dc5

lostmsu added the rfc label Feb 13, 2020

lostmsu commented Feb 13, 2020

View reviewed changes

lostmsu added 2 commits February 13, 2020 15:51

BorrowedReference.Pointer is a private readonly field

7304b25

drop C# 8.0 requirement

2bc218c

lostmsu requested a review from filmor February 14, 2020 00:05

lostmsu mentioned this pull request Feb 20, 2020

Modernisation #1015

Closed

amos402 reviewed Feb 23, 2020

View reviewed changes

lostmsu added 2 commits February 23, 2020 11:12

renamed NewReference.ToPyObject to MoveToPyObject

2cf9fc3

removed public property Pointer from NewReference and replaced with DangerousGetAddress

added xmldoc comments to *Reference members

6ae63cd

filmor merged commit 770fc01 into pythonnet:master Feb 26, 2020

amos402 mentioned this pull request Mar 10, 2020

Add PyHandle #1087

Closed

4 tasks

Proposal: Safe pointers #1043

Proposal: Safe pointers #1043

Uh oh!

Conversation

lostmsu commented Feb 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this implement/fix? Explain your changes.

Downsides

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lostmsu commented Feb 14, 2020

Uh oh!

codecov-io commented Feb 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

amos402 commented Feb 14, 2020

Uh oh!

lostmsu commented Feb 22, 2020

Uh oh!

lostmsu commented Feb 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lostmsu commented Feb 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filmor commented Feb 26, 2020

Uh oh!

Uh oh!

lostmsu commented Feb 13, 2020 •

edited

Loading

codecov-io commented Feb 14, 2020 •

edited

Loading

lostmsu commented Feb 26, 2020 •

edited

Loading