add a scope class to manage the context of interaction with Python an… #381

yagweb · 2017-02-15T16:44:22Z

It‘s a prototype of this proposal here to help its discussion.

Modifications are located at file pythonengine.cs, A class named PyScope was added for the variable management and exchanging between .NET and Python.

Several unit tests were given in a new file "src/embed_tests/pyscope.cs" which can be used as examples.

One of the example,

using (Py.GIL())
{
    dynamic ps = Py.CreateScope();
    ps.Set("bb", 100);         //declare a variable
    ps.cc = 10;                  //declare a variable dynamically
    int aa = ps.Eval<int>("bb+cc+3");     //113

    //add a function to the scope, and call it
    //global keyword was used here to show what pyscope can do 
    ps.aa = 0; 
    ps.Exec(
        "def func1():\n" +
        "    global aa \n" +
        "    aa = bb+cc+3 \n"            
    );
    ps.func1(); //call the function
    int aa1 = ps.Get<int>("aa");  //get variable
    int aa2 = ps.aa.As<int>();     //get variable dynamically

    //create a new scope and import the variables
    PyScope ps2 = ps.CreateScope();
    ps2.ImportModule("numpy", "np");
    var aa3 = ps2.Eval<float>("np.sin(bb)");
}

vmuriart · 2017-02-15T17:30:24Z

src/runtime/pythonengine.cs

@@ -439,7 +454,7 @@ public static PyObject ModuleFromString(string name, string code)
                borrowedLocals = false;
            }

-            var flag = (IntPtr)257; /* Py_file_input */
+            var flag = (IntPtr)_flag; /* Py_file_input */


The comment here should be updated

vmuriart · 2017-02-15T17:42:35Z

src/embed_tests/pyscope.cs

+        }
+
+        /// <summary>
+        /// Set the attribute of a pyobject to null.


Is comment right?

yagweb · 2017-02-17T04:49:15Z

@vmuriart These two comments are updated.

den-run-ai · 2017-02-18T20:26:45Z

@yagweb can you update docs or readme with few examples?

yagweb · 2017-02-19T05:57:43Z

@denfromufa The unit tests in "src/embed_tests/pyscope.cs" are commented and can be used as examples.

Currently the relationships among PythonEngine, Py, PyScope, Py.GILState, PyScope.GILState are a little messy. The APIs may changes at any time, so I may not add these examples into docs or readme until the APIs become stable. We can use the unit tests instead.

vmuriart · 2017-02-20T23:27:58Z

@yagweb should be clarify the intents/differences of them before adding the feature? I agree on not adding it to the documented API until we hash it out though.

den-run-ai · 2017-02-21T03:44:55Z

@yagweb @vmuriart I'm still struggling to understand use cases for scopes, except for storing local state.
Maybe I'm missing something? That's why I asked to add something in the docs.

yagweb · 2017-02-23T16:04:25Z

@vmuriart I am sorry for that, the modifications are too many last time. Last time, the PR is cleaned up by separating a new PR ‘add eval and exec’ out from it, merging two classes, renaming methods, etc. I updated all my comments and forgot to inform you. In the future, I will provide detailed comments for each commit.

@denfromufa @vmuriart Storing local state is the start point. I wanted to embedding python into a C# application using pythonnet, and found the APIs for this scene are hard to use. In my another comment here, I concluded that

A scope class aims at managing the context of a interaction with Python and simplify the variable exchanging. It can act like a Python Module in .NET. and can be shared. In the Scope class, many python functions can be wrapped to operate with the local variables and global variables for easy-to-use and speeding up, such as the ImportAs method.

I just make a new commit, now this PR implements almost all the functions of IronPython about accessing Python code from .NET code. I still don't know whether it's enough or not until it's actually used.

yagweb · 2017-02-23T16:44:15Z

@denfromufa @vmuriart The most complicated thing about using pythonnet in .NET application is the GIL. This is the cause of what I said,

Currently the relationships among PythonEngine, Py, PyScope, Py.GILState, PyScope.GILState are a little messy.

Here is a related proposal, but I think it's impossible in some scenes.

I think there are four scenes when embedding CPython.

If the application is single thread, it's no need to AcquireLock and ReleaseLock all the time, we can AcquireLock when initialize and ReleaseLock when Shutdown.
In a multi-thread application, if several fast execution python functions need to called, we can treat the CPython as a shared resource, and the GIL as a lock in .NET. So a solution is wrapping each function with AcquireLock and ReleaseLock, this is what the pythonnet does right now.
In a multi-thread application, if the python functions needs waiting IO and very time consuming. Then the GIL cannot be managed in .NET code. The python threading can be used and the .NET code should use asynchronous calls to interacting with Python. A similar use case is the Dispatcher of WPF.
In a multi-thread application, if multi-cores are needed. Use CPython with multi-processing, or use pypy and IronPython instead.

In the multi-thread scenes, it's better to use IronPython if possible. In the 3rd scene, I think acquire GIL by default in .NET is impossible.

den-run-ai · 2017-02-23T19:27:59Z

@yagweb in my view the multi-threading and asynchronous execution can be outside of this PR. GIL issues are well-known in Python community. Besides multi-processing in CPython you can look at Cython with nogil contexts, and Numba with nopython mode for numerical code. PyPy-STM is not ready for production use and only available on Linux. PyPy with cpyext is currently crashing when importing pythonnet:

#330

vmuriart · 2017-02-24T00:52:57Z

Even for the 1st scene the gc and Finalize in classderived.cs might throw a wrench into things.

Looks like might need to rebase this one since #389 was merged in.

yagweb · 2017-02-24T08:24:49Z

@denfromufa I see, thank you.
When users need multi-core parallel in embedded CPython, they should design their code with any means including Cython, Numba etc. It’s out of pythonnet. So the 4th scene can be deleted from this list.

@vmuriart Thank you. it seems like there no "real" single-threaded application since the GC working behind and needing to call CPython. So the 1st scene can also be deleted from this list.

@denfromufa @vmuriart May it be better to make the PyScope class independent from the GIL solution?

codecov · 2017-02-24T09:12:47Z

Codecov Report

Merging #381 into master will decrease coverage by 0.21%.
The diff coverage is 58.79%.

@@            Coverage Diff             @@
##           master     #381      +/-   ##
==========================================
- Coverage   68.56%   68.34%   -0.22%     
==========================================
  Files          63       64       +1     
  Lines        5344     5560     +216     
  Branches      855      894      +39     
==========================================
+ Hits         3664     3800     +136     
- Misses       1444     1496      +52     
- Partials      236      264      +28

Flag	Coverage Δ
#setup_linux	`75.71% <ø> (ø)`	⬆️
#setup_windows	`67.68% <58.79%> (-0.2%)`	⬇️

Impacted Files	Coverage Δ
src/runtime/runtime.cs	`88.14% <ø> (ø)`	⬆️
src/runtime/pythonengine.cs	`78.69% <100%> (+0.86%)`	⬆️
src/runtime/pyobject.cs	`37.71% <20%> (+2.35%)`	⬆️
src/runtime/pyscope.cs	`57.92% <57.92%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce14424...b9dcdac. Read the comment docs.

den-run-ai · 2017-02-27T18:27:08Z

@denfromufa @vmuriart May it be better to make the PyScope class independent from the GIL solution?

I think it is safer that GIL is held inside PyScope by default. You can maybe pass non-default argument to release the GIL.

vmuriart · 2017-02-27T19:49:58Z

@yagweb just finished reviewing this pr and makes sense what you're proposing. I think there's some refactoring we can do to clean up the code and there's a couple tests I want to run (like what happens if we create two PyScope instead of a PyScope and a SubScope) (does changing a variable in the subscope that exists on the parent scope modify it).

vmuriart · 2017-02-27T20:35:53Z

Looks like there is a gc issue lingering somewhere.

https://travis-ci.org/pythonnet/pythonnet/jobs/204901628#L1606

den-run-ai · 2017-02-27T20:56:47Z

@vmuriart how about adding this?

#400 (comment)

vmuriart · 2017-02-27T21:12:53Z

src/runtime/pythonengine.cs

+            {
+                return;
+            }
+            Runtime.XDecref(globals);


gc issue is most likely here. You'll need to check that the engine is initialize and that it isn't finalizing before running these.

vmuriart · 2017-02-27T21:14:44Z

That wouldn't fix it. The issue is that the engine get shutdown earlier at "some point" and looks like currently there is nothing checking the status of the engine before trying to run more commands on it.

vmuriart · 2017-02-28T16:34:06Z

Rebased and updated some of the internal changes

yagweb · 2017-04-27T11:07:22Z

@denfromufa One of @filmor's suggestions is,

The function names for accessing the variables are a bit too verbose for my taste, Get and Set would be enough.

My response is,

I agreed the names are too verbose, but these names and the Exception messages all came from IronPython. Just to be compatible.

Personally I also prefer to the shorter names Get, Set, Remove and Contains.

In addition, Execute can be renamed to Exec, AddVariables can be renamed to ImportDict. Or maybe the ImportDict, ImportModule, and ImportScope can all be renamed to Import.

So, compatible with IronPython or more concise, what's your opinion?

JimSEOW · 2017-04-27T11:15:41Z

@yagweb will the scope class address the GET, SET issues discussed here

yagweb · 2017-04-27T13:27:07Z

@JimSEOW I put my modifications here. If your have any other questions, please talk it there.

den-run-ai · 2017-04-29T04:30:26Z

@yagweb regarding your previous comment I would go for more concise, @tonyroberts @vmuriart @filmor do you agree?

JimSEOW · 2017-04-29T09:13:46Z

@yagweb @denfromufa etc.
When Xamarin create .NET binding to C++ libraries, they try their best to .dotNETfy the binding using semantics that are characteristics to the .NET ecosystem.
Perhaps we can do the same here.
=> Get and Set are more the typical Java syntax for properties (not sure how easy it is to replace them with alternatives)
=> I prefer Execute to Exec
=> We are addressing users from both .NET and python ecosystems. It will be interesting to strike a balance. However, we are talking about Python 4 .NET here. What can we learn from Ironpython to suggest some naming suggestions here?

den-run-ai · 2017-05-24T19:55:51Z

@filmor @vmuriart I would like to use this API during my upcoming presentation on June 8th at Microsoft. Please review and comment, if not accepting!

vmuriart · 2017-05-25T17:05:37Z

I'll review over the weekend. I don't think this one introduced any GC bugs but im seeing a bunch of warnings showing up on the Python Tests now (though I suspect its unrelated to this but want to check).

I'll review @filmor comments to make sure requests were resolved.

filmor · 2017-04-27T07:55:20Z

src/embed_tests/TestPyScope.cs

@@ -170,7 +170,7 @@ public void TestImportModule()

                ps.Exec("sys.attr1 = 2");
                var value1 = ps.Eval<int>("sys.attr1");
-                var value2 = (int)sys.attr1.AsManagedObject(typeof(int));
+                var value2 = (int)sys.attr1.As<int>();


Remove the cast.

filmor · 2017-04-27T07:56:30Z

src/embed_tests/TestPyScope.cs

+                var result = scope.GetVariable<int>("aa");
+                Assert.AreEqual(113, result);
+
+                scope.Dispose();


Use using to give a good example?

filmor · 2017-04-27T07:57:56Z

src/runtime/pyscope.cs


-        private bool isDisposed;
+        internal readonly IntPtr obj;


Document the internal fields.

filmor · 2017-04-27T07:58:53Z

src/runtime/pyscope.cs


-        internal PyScope(string name)
+        private bool isDisposed;


Private should be _isDisposed.

filmor · 2017-04-27T07:59:17Z

src/runtime/pyscope.cs

+            {
+                name = "";
+            }
+            var module = Runtime.PyModule_New(name);


Very nice :)

i'm still pondering what did you notice here so special @filmor? 🤔

filmor · 2017-05-30T14:53:07Z

src/runtime/pyscope.cs

+            {
+                value = default(T);
+                return true;
+            }


This means that if someone does TryGetVariable<int>("something", out val) where something = None, they will get as a result val = 0 without any further indication. Seems like a very bad idea for value types.

filmor · 2017-05-30T14:54:44Z

src/runtime/pyscope.cs

+            }
+            if (name != null && NamedScopes.ContainsKey(name))
+            {
+                throw new PyScopeException($"PyScope '{name}' has existed");


"A scope of name '{name}' does already exist"

filmor · 2017-05-30T14:55:29Z

src/runtime/pyscope.cs

+            {
+                throw new ArgumentNullException(nameof(name));
+            }
+            if (name != null && NamedScopes.ContainsKey(name))


Contains(name)

Sorry, I'm not quite understand?

You already defined a member function Contains, use that one.

filmor · 2017-05-30T14:56:40Z

src/runtime/pyscope.cs

+            {
+                return NamedScopes[name];
+            }
+            throw new PyScopeException($"PyScope '{name}' not exist");


Maybe "There is no scope named '{name}' registered in this manager"?

filmor · 2017-05-30T14:57:11Z

src/runtime/pyscope.cs

+        {
+            PyScope value;
+            NamedScopes.TryGetValue(name, out value);
+            return value;


As said above, just forward the semantics of TryGetValue directly.

filmor · 2017-05-30T15:03:58Z

Regarding doing things like IronPython: I doubt that we actually want API parity on the .NET side, for that we are too far away. I also don't see much of a benefit, pythonnet is not built on top of the scripting architecture used by IronPython but directly on Python's API, so they can not be replacements for each other anyhow.

This is completely disconnected from the discussion how the Python side should look like.

yagweb · 2017-05-31T17:16:22Z

@filmor Very thank you for your detailed and valuable comments.

@denfromufa @filmor @vmuriart The last question is, should the methods 'GetVariable', 'SetVariable' etc. be renamed to shorter ones as 'Get', 'Set' etc.?

den-run-ai · 2017-05-31T18:06:32Z

@yagweb i like the shorter names Get & Set.

yagweb · 2017-06-01T01:06:54Z

The names 'Get & Set' etc. make sense, because variables are the only content a scope managed.
An example of this design is the 'Add' method of the List.

Methods are renamed and documents are added.

den-run-ai · 2017-06-06T07:30:08Z

@vmuriart i'm planning to merge this on Wednesday, let us know if you have any feedback before that. @yagweb great work, thanks for enduring so many iterations of reviews.

pythonnet#381) * add a scope class to manage the context of interaction with Python and simplify the variable exchanging * Deprecate public RunString Had to remove defaults to disambiguate call on `internal RunString`. Can re-add after removing `public RunString` Closes pythonnet#401 * Rename several methods and add three methods Referring to IronPython, change the name of the methods Get, Exists, SetLocal, DelLocal to GetVariable, ContainsVariable, SetVariable, RemoveVariable. Hidden the methods SetGlobalVariable, RemoveGlobalVariable. Add a new method 'Compile' to compile string into ast, the ast can be seen as the ScriptSource of IronPython. Add two new methods 'Execute' and 'Execute<T>' to execute an ast and obtain the result, corresponding to the 'Execute' method of IronPython. * rebased * Rebased update * format cleanup * create unnamed pyscope, make PyScope.GILState save remove method GetInstHandle add function to create unnamed pyscope add a field isDisposed for PyScope.GILState to make it more save * fixup! create unnamed pyscope, make PyScope.GILState save * remove GIL and rebased * Add several methods add ImportScope: a scope can import variable from any scope, equivalent to python 'import * from mod' add dynamic member support add an OnDispose event remove the field ‘globals’ referring to python module put the scope class in a new file Unit test: TestThread uses scope function replacing Exec/Eval to speed up the execution. * add a Variables method * fixup! add a Variables method * remove private method _GetVariable * add unit test for Variables() method * add several methods and rebased Add an optional dict parameter for Eval/Exec/Execute methods Add a new field obj which point to a Python Module (same as pyobject) Add a static method New Rename the old ImportScope method to ImportAllFromScope Add a new ImportScope method and add a unit test for it Rename the CreateScope method to NewScope * add a new class PyScopeManager * cleaned up the Import methods * updated according to filmor's comments * fixup! updated according to filmor's comments * Get/Set Methods renamed

yagweb mentioned this pull request Feb 15, 2017

Add a class to manage the context of a interaction with Python and simplify the variable exchanging #380

Closed

yagweb force-pushed the add_scope branch from eb6c17b to f7036d7 Compare February 15, 2017 17:00

vmuriart reviewed Feb 15, 2017

View reviewed changes

src/embed_tests/pyscope.cs Outdated

}

/// <summary>

/// Set the attribute of a pyobject to null.

Copy link

Contributor

vmuriart Feb 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is comment right?

yagweb force-pushed the add_scope branch from 82c21d7 to c65d1d0 Compare February 17, 2017 08:38

yagweb force-pushed the add_scope branch from 69598cf to f413784 Compare February 19, 2017 04:16

yagweb force-pushed the add_scope branch from 354fdd4 to 4508a02 Compare February 19, 2017 07:21

vmuriart force-pushed the master branch from aef30c1 to 3212cfe Compare February 20, 2017 22:45

yagweb force-pushed the add_scope branch from 9981756 to f9d047e Compare February 24, 2017 09:12

yagweb mentioned this pull request Feb 24, 2017

Deprecate Public RunString #401

Closed

vmuriart reviewed Feb 27, 2017

View reviewed changes

vmuriart force-pushed the add_scope branch from f9d047e to 46b6cda Compare February 28, 2017 16:23

yagweb mentioned this pull request Feb 28, 2017

embedding docs - expose all available API #410

Closed

cleaned up the Import methods

497d7aa

denfromufa added 2 commits May 24, 2017 11:28

Merge branch 'master' into add_scope

60ce28b

Merge branch 'master' into add_scope

ebf5a2b

den-run-ai mentioned this pull request May 26, 2017

RunSimpleString variarible not visible in Eval #474

Closed

filmor requested changes May 30, 2017

View reviewed changes

filmor added waiting for user feedback and removed requires review labels May 30, 2017

updated according to filmor's comments

2c3db3d

yagweb force-pushed the add_scope branch from d9ebcf3 to 2c3db3d Compare May 31, 2017 17:00

fixup! updated according to filmor's comments

30543eb

Get/Set Methods renamed

b9dcdac

den-run-ai closed this Jun 6, 2017

den-run-ai reopened this Jun 6, 2017

filmor approved these changes Jun 6, 2017

View reviewed changes

den-run-ai removed the waiting for user feedback label Jun 6, 2017

den-run-ai merged commit 550a027 into pythonnet:master Jun 9, 2017

yagweb deleted the add_scope branch June 9, 2017 23:33

add a scope class to manage the context of interaction with Python an… #381

add a scope class to manage the context of interaction with Python an… #381

Conversation

yagweb commented Feb 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yagweb commented Feb 17, 2017

den-run-ai commented Feb 18, 2017

yagweb commented Feb 19, 2017 • edited Loading

vmuriart commented Feb 20, 2017

den-run-ai commented Feb 21, 2017

yagweb commented Feb 23, 2017 • edited Loading

yagweb commented Feb 23, 2017 • edited Loading

den-run-ai commented Feb 23, 2017

vmuriart commented Feb 24, 2017

yagweb commented Feb 24, 2017

codecov bot commented Feb 24, 2017 • edited Loading

Codecov Report

den-run-ai commented Feb 27, 2017

vmuriart commented Feb 27, 2017

vmuriart commented Feb 27, 2017

den-run-ai commented Feb 27, 2017

Choose a reason for hiding this comment

vmuriart commented Feb 27, 2017

vmuriart commented Feb 28, 2017

yagweb commented Apr 27, 2017 • edited Loading

JimSEOW commented Apr 27, 2017

yagweb commented Apr 27, 2017

den-run-ai commented Apr 29, 2017

JimSEOW commented Apr 29, 2017 • edited Loading

den-run-ai commented May 24, 2017

vmuriart commented May 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filmor commented May 30, 2017

yagweb commented May 31, 2017

den-run-ai commented May 31, 2017

yagweb commented Jun 1, 2017

den-run-ai commented Jun 6, 2017

yagweb commented Feb 15, 2017 •

edited

Loading

yagweb commented Feb 19, 2017 •

edited

Loading

yagweb commented Feb 23, 2017 •

edited

Loading

yagweb commented Feb 23, 2017 •

edited

Loading

codecov bot commented Feb 24, 2017 •

edited

Loading

yagweb commented Apr 27, 2017 •

edited

Loading

JimSEOW commented Apr 29, 2017 •

edited

Loading