Multi-context execution (py.Context) #144

drew-512 · 2021-12-10T18:27:15Z

Hi Seb, it's a been a couple months but things are well here.

With this PR, I am pleased to share gpython with multi-context execution! Check out type Context interface and vm/vm_test.go for a quick start.

My intention with this PR is get feedback from you on what you want me to alter or massage so that you are satisfied. I am totally open if you want me to shift, rename, or organize code in ways that you would prefer, so kindly please just let me know. I think many will be highly interested in a multi-ctx gpython (as many have asked for it as you already know).

So if you like things as they are, you can merge this PR. Otherwise, just give me a list of things you want changed and I'll resubmit a new PR with that. Or, perhaps merge this PR as is, insert edits as you see fit, tag it into a new release and version, and then I will follow your lead and ditch my fork and we can be on our way. :)

We are definitely using gpython here for our classified particle research project and would love to join forces with you on maintaining gpython. I think it is highly reasonable, especially with this PR, that gpython is picked up (sponsored) since it serves the Google, Python, and Go communities quite uniquely.

drew-512 · 2021-12-10T18:53:40Z

Liking the your CI scripts!

Added two more commits that addressed those tests not passing

sbinet · 2021-12-13T10:01:13Z

thanks. it's quite a large PR :)
I'll try to digest it over the coming days (with probably a hiatus Tue-Wed).

drew-512 · 2021-12-13T16:00:39Z

Hi friend. You'll see that much of the code is just additional helper/utility code (as well as the couple inner core changes relating to py.Ctx). I am open on any renaming or code reorganization that calls to you. I would like to build trust with you and continue to grow gpython, so please let me know on what flow and communication modes you prefer. If you are feeling ambitious, the code now is in a position for us to put better import code in or many other kinds of things that people want. A multi-ctx python in Go will be fo interest to many and I think gpython has major potential, as it bridges the gap between devs like us plugged into the power of Go and the kind of people we make software for (scientists and researchers and UI devs who are best with Python for their work).

In the coming months, I will be using gypthon to power a Go-based UI runtime that "serves" a Unity client UI. The Unity client runs a 2D physics-based UI and embeds the Go process. This will power the UI for this particle research suite we are developing...

sbinet

a very first pass: I see there are a bunch of files which have a 0644 -> 0755 chmod change.
could this be reversed?
(I don't think this brings anything to these files. or does it?)

a naive question: do we expect multiple py.Ctx implementations? wouldn't be easier to start with just a concrete Ctx in py?

also: there are a couple of fixes which are useful to have at some point but aren't needed for the multi-context change. could these be part of another PR?

thanks.

go.sum

modules/runtime.go

py/frame.go

py/function.go

py/import.go

py/module.go

py/run.go

py/util.go

vm/vm_test.go

modules/runtime.go

sbinet · 2021-12-16T17:57:30Z

also, it would be great to have an idea of the loss (probably?) of performances wrt "single context" execution.

Co-authored-by: Sebastien Binet <[email protected]>

drew-512 · 2022-01-13T16:51:44Z

Love all the CI stuff you're adding, awesome!

The items I'm working on in the coming days are a README rev, a well documented embedded example (huge for newcomers), and the single vs multi ctx benchmark as you suggested. I think gpython as a viable, reliable embedded interpreter solution has very high potential. Something as potent, easy, and fast to embed as Lua, but in general, devs don't ever want to choose that b/c they don't want to tell their users to have to learn {{$esoteric_embedded_language}}. My vision is that RunFile(), multi-context, and this embedding example are inviting and positive signs for those evaluating gpython for serious use.

I've heard many big cats often quip "the second most popular language in the world", reflecting that when it comes to non-developers (finance, science), Python has the best learning curve and those users typically already have some python experience. Also, many of those types want to grow their python skills b/c they see it show up all over their career. So I suspect this could be a big theme, now we are in world awash in interpreters that are basically only interesting to professional developers. The gpython multi-context additions for this physics research group I mentioned are working out very well and seem to "complete" gpython for real world Go applications.

I would love to remind you and @ncw what a huge freaking achievement running any python 3 in Go is, and that gpython's potential is not to be underestimated as a leading Go embedding option. Or perhaps I see no reason why Google would not want to donate a summer's worth of resources to gpython if that is something beneficial to gpython.

Drew

sbinet

thanks for the update.

but, before going further, could we try to address the "single-thread before/after-context" benchmarking issue?

then I'd like to merge this PR in, so as to reduce the amount of reviewing work.
thanks a lot.

modules/runtime.go

py/run.go

sbinet · 2022-01-19T09:41:40Z

modules/runtime.go

+	store     *py.ModuleStore
+	opts      py.ContextOpts
+	closeOnce sync.Once
+	closing   bool


is closing that useful?
it doesn't seem to be really used as of yet.

The intent is that Close() can be use reliably and easily, so multiple concurrent Close() support seems important. And when it comes at no allocation cost, that makes sense to me.

I am not disputing the fact that Close() is useful, but AFAICT closing is accessed twice (once initialized with false, then modified to true) but its value isn't used much otherwise.

drew-512 · 2022-01-19T21:44:13Z

Indeed! :D

Good suggestions, btw.

Commits coming in by the end of this weekend is a documented, real-world embedding example, readme edits, and that benchmark.

drew-512 · 2022-01-19T21:45:50Z

I'll switch to do and commit that benchmark first -- just had been putting that off, waaah.

drew-512 · 2022-01-21T17:32:47Z

@sbinet for the pre-context vs with-context benchmarks you raised, I'm not clear on what the path is. I am getting that you want a PR that does bench X on gpython before py.Context and then see how the test performs with the py.Context applied.

Questions:

Is the intention/purpose to just see if there's a load penalty of an extra interface{} traveling around everywhere? I mean, I can't see how it would be anything but negligible, especially when you look at how much GC load there is from any kind of execution. Really, the only change is that instead of going to a global module lookup, the lookup occurs on a context-instanced map which of course is lateral.
Because thepy.Context is just another field in the VM struct, I'm not even sure how any python code can/would test any performance hit, other than, say, comparing the pystone of pi_chudnovsky_bs.py before and after the py.Context adds. In your prev post, you mentioned a "local" variable edit -- do you mean embedded callbacks or editing a module local?

Thanks.

drew-512 · 2022-01-26T16:49:08Z

Hi everyone!

At this point I am holding off on any new work on my fork and awaiting guidance on anything further. I am happy to take feedback and @sbinet please let me know on my questions above regarding bench comparison. In summary: analysis of the py.Context shows, at most, the penalty can't even be measured.

Aside from readme or embedded example edits, I recommend we merge and celebrate this exciting step forward. A lot of people will find this multi-context capability exciting and recognize what it unlocks!

drew-512 · 2022-01-26T17:04:04Z

Big thanks to everyone has helped make gpython happen over the years -- though it has been some time!

@ncw it would be great to get any feedback or your blessing on any of the code or the README.

Otherwise, @sbinet has been making great suggestions and I think we're in great shape to almost ready to merge! 🎉

sbinet

last nitpicks.

I'd prefer to keep:

7d2a8d9 (print kwargs)
ba7db80 (kwargs)
61e42c8 (embedding)
out of this PR (and have those in dedicated PRs)

also, it would be great if you could create a dedicated branch for these PRs (easier on the review process).

could you send another PR against go-python/licence adding yourself to the AUTHORS and/or CONTRIBUTORS file?
(then, once the multi-context PR is validated, I'll merge the licence one and the multi-context one)

thanks.

sbinet · 2022-01-27T11:59:03Z

.vscode/launch.json

@@ -0,0 +1,26 @@
+{


I'd remove this file.

README.md

sbinet · 2022-01-27T12:05:31Z

README.md

+The [embedding example](https://github.com/go-python/gpython/tree/master/examples/embedding) demonstrates how to 
+easily embed and invoke gpython from any Go application.


I'd rather prefer having the embedding stuff be part of another PR.

py/util.go

sbinet · 2022-01-27T12:09:44Z

py/util.go

+	return nil
+}
+
+// LoadIntsFromList extracts a list of ints contained given a py.List or py.Tuple


Suggested change

// LoadIntsFromList extracts a list of ints contained given a py.List or py.Tuple

// LoadIntsFromList extracts a list of ints contained in the provided py.List or py.Tuple

py/util.go

sbinet · 2022-01-27T12:24:13Z

modules/runtime.go

+	store     *py.ModuleStore
+	opts      py.ContextOpts
+	closeOnce sync.Once
+	closing   bool


I am not disputing the fact that Close() is useful, but AFAICT closing is accessed twice (once initialized with false, then modified to true) but its value isn't used much otherwise.

Co-authored-by: Sebastien Binet <[email protected]>

sbinet · 2022-02-07T14:30:04Z

superseded by #158.

Drew O'Meara added 15 commits December 7, 2021 21:07

py.CompileMode now a string enum

fc66b7f

addressed benign Go warnings

d10c35f

make Float implement M__index__

2161b71

added string helper functions

247ec42

updated to 1.15

9a75271

new Ctx-aware module model

90a8988

py.Ctx model allows concurrent execution

c51644e

repl.New() now accepts an existing py.Ctx

7f7d213

fixed typo in callInternal()

708bb7e

unbroke test compatibilty

a244256

cleaned up py.Ctx initialization wrt sys and DefaultCtxOpts()

75d8743

added multi Ctx benchmark

d5fb8e8

WIP

269bdc0

fixed py.NewCtx not being set

4db9c65

fixed List.sort() detection of instance invocation

53d05db

separated path resolution from path evaluation

7fd1c29

sbinet requested a review from ncw December 13, 2021 09:58

sbinet reviewed Dec 16, 2021

View reviewed changes

modules/runtime.go Show resolved Hide resolved

Drew O'Meara and others added 7 commits December 18, 2021 17:24

removed cruft

8f3a104

place ctx at top

5ebaf66

err msg tweak

d423190

Co-authored-by: Sebastien Binet <[email protected]>

typo fix

b33c266

Co-authored-by: Sebastien Binet <[email protected]>

for loops we can all agree on

efca761

Co-authored-by: Sebastien Binet <[email protected]>

removed outdated comments

1569475

Merge branch 'master' of https://github.com/drew-512/gpython

a9c5f1f

drew-512 changed the title ~~Multi-context execution (py.Ctx)~~ Multi-context execution (py.Context) Jan 13, 2022

added Context.Close() and ModuleImpl.OnContextClosed()

681abf2

sbinet reviewed Jan 19, 2022

View reviewed changes

docs and LoadIntsFromList

7508556

Drew O'Meara added 7 commits January 21, 2022 12:38

rename edits

99bcb40

push/popBusy now private

0cc7650

doc edits

9afb2d3

fixed kwarg issues in ParseTupleAndKeywords() and builtin_print test

7d2a8d9

added Flush kwarg to print test

ba7db80

added embedding example

61e42c8

main README makeover

168c337

This was referenced Jan 26, 2022

The problem implementing print in gpython. #91

Closed

Is it safe to use gpython interpreter in concurrent processing? #132

Closed

fixed type conversion

2b7e1ab

sbinet reviewed Jan 28, 2022

View reviewed changes

drew-512 and others added 7 commits January 28, 2022 09:40

Update README.md

3821b4b

Co-authored-by: Sebastien Binet <[email protected]>

Update README.md

170f0d2

Co-authored-by: Sebastien Binet <[email protected]>

Update py/util.go

b68b9f6

Co-authored-by: Sebastien Binet <[email protected]>

Update py/util.go

759023b

Co-authored-by: Sebastien Binet <[email protected]>

Update py/util.go

0c77716

Co-authored-by: Sebastien Binet <[email protected]>

LoadIntsFromList cleanup

505755d

comment edits

02d6b3e

sbinet closed this Feb 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-context execution (py.Context) #144

Multi-context execution (py.Context) #144

drew-512 commented Dec 10, 2021 •

edited

Loading

drew-512 commented Dec 10, 2021

sbinet commented Dec 13, 2021

drew-512 commented Dec 13, 2021

sbinet left a comment

sbinet commented Dec 16, 2021 •

edited

Loading

drew-512 commented Jan 13, 2022 •

edited

Loading

sbinet left a comment

sbinet Jan 19, 2022

drew-512 Jan 21, 2022

sbinet Jan 27, 2022

drew-512 commented Jan 19, 2022 •

edited

Loading

drew-512 commented Jan 19, 2022

drew-512 commented Jan 21, 2022

drew-512 commented Jan 26, 2022 •

edited

Loading

drew-512 commented Jan 26, 2022

sbinet left a comment

sbinet Jan 27, 2022

sbinet Jan 27, 2022

sbinet Jan 27, 2022

sbinet Jan 27, 2022

sbinet commented Feb 7, 2022

		The [embedding example](https://github.com/go-python/gpython/tree/master/examples/embedding) demonstrates how to
		easily embed and invoke gpython from any Go application.

	// LoadIntsFromList extracts a list of ints contained given a py.List or py.Tuple
	// LoadIntsFromList extracts a list of ints contained in the provided py.List or py.Tuple

Multi-context execution (py.Context) #144

Multi-context execution (py.Context) #144

Conversation

drew-512 commented Dec 10, 2021 • edited Loading

drew-512 commented Dec 10, 2021

sbinet commented Dec 13, 2021

drew-512 commented Dec 13, 2021

sbinet left a comment

Choose a reason for hiding this comment

sbinet commented Dec 16, 2021 • edited Loading

drew-512 commented Jan 13, 2022 • edited Loading

sbinet left a comment

Choose a reason for hiding this comment

sbinet Jan 19, 2022

Choose a reason for hiding this comment

drew-512 Jan 21, 2022

Choose a reason for hiding this comment

sbinet Jan 27, 2022

Choose a reason for hiding this comment

drew-512 commented Jan 19, 2022 • edited Loading

drew-512 commented Jan 19, 2022

drew-512 commented Jan 21, 2022

drew-512 commented Jan 26, 2022 • edited Loading

drew-512 commented Jan 26, 2022

sbinet left a comment

Choose a reason for hiding this comment

sbinet Jan 27, 2022

Choose a reason for hiding this comment

sbinet Jan 27, 2022

Choose a reason for hiding this comment

sbinet Jan 27, 2022

Choose a reason for hiding this comment

sbinet Jan 27, 2022

Choose a reason for hiding this comment

sbinet commented Feb 7, 2022

drew-512 commented Dec 10, 2021 •

edited

Loading

sbinet commented Dec 16, 2021 •

edited

Loading

drew-512 commented Jan 13, 2022 •

edited

Loading

drew-512 commented Jan 19, 2022 •

edited

Loading

drew-512 commented Jan 26, 2022 •

edited

Loading