Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Multi-context execution (py.Context) #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 81 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
fc66b7f
py.CompileMode now a string enum
Dec 8, 2021
d10c35f
addressed benign Go warnings
Dec 8, 2021
2161b71
make Float implement M__index__
Dec 8, 2021
247ec42
added string helper functions
Dec 8, 2021
9a75271
updated to 1.15
Dec 8, 2021
90a8988
new Ctx-aware module model
Dec 9, 2021
c51644e
py.Ctx model allows concurrent execution
Dec 9, 2021
7f7d213
repl.New() now accepts an existing py.Ctx
Dec 9, 2021
708bb7e
fixed typo in callInternal()
Dec 9, 2021
a244256
unbroke test compatibilty
Dec 9, 2021
75d8743
cleaned up py.Ctx initialization wrt sys and DefaultCtxOpts()
Dec 10, 2021
d5fb8e8
added multi Ctx benchmark
Dec 10, 2021
269bdc0
WIP
Dec 10, 2021
4db9c65
fixed py.NewCtx not being set
Dec 10, 2021
53d05db
fixed List.sort() detection of instance invocation
Dec 10, 2021
7fd1c29
separated path resolution from path evaluation
Dec 11, 2021
8f3a104
removed cruft
Dec 18, 2021
5ebaf66
place ctx at top
Dec 18, 2021
d423190
err msg tweak
drew-512 Dec 18, 2021
b33c266
typo fix
drew-512 Dec 18, 2021
efca761
for loops we can all agree on
drew-512 Dec 18, 2021
1569475
removed outdated comments
Dec 18, 2021
a9c5f1f
Merge branch 'master' of https://github.com/drew-512/gpython
Dec 18, 2021
6372aee
added info for Frame.Ctx
Dec 19, 2021
4af4f90
removed unnecessary assignment
Dec 19, 2021
aec59f8
copyright and formatting
Dec 19, 2021
74f1a19
Ctx and NewCtx docs
Dec 19, 2021
8fa6622
switch cleanup
Dec 19, 2021
26fb9aa
leaner ModuleImpl schema
Dec 20, 2021
8614aeb
added GetDict() impl
Dec 20, 2021
86531c6
Compile() now returns py.Code (vs py.Object)
Dec 20, 2021
f71b961
tighter py.Ctx invocation patterns
Dec 20, 2021
77a7954
WIP
Dec 20, 2021
6bba7da
reverted to 644
Dec 20, 2021
46d1052
chmod 644
Dec 20, 2021
66fad23
enhanced RunFile() to enclose RunInNewModule()
Dec 27, 2021
95afd7b
Update examples/multi-ctx/main.go
drew-512 Jan 10, 2022
319edb7
Update modules/runtime.go
drew-512 Jan 10, 2022
06a2eae
Update modules/runtime.go
drew-512 Jan 10, 2022
8d4d586
Update modules/runtime.go
drew-512 Jan 10, 2022
d475e67
Update vm/eval.go
drew-512 Jan 10, 2022
6afac99
Update py/run.go
drew-512 Jan 10, 2022
931c346
Update modules/runtime.go
drew-512 Jan 10, 2022
a7ce4bb
Update modules/runtime.go
drew-512 Jan 10, 2022
c1f8809
Update modules/runtime.go
drew-512 Jan 10, 2022
9f02cf1
Update modules/runtime.go
drew-512 Jan 10, 2022
85e3a4a
Update modules/runtime.go
drew-512 Jan 10, 2022
fbec34e
Update modules/runtime.go
drew-512 Jan 10, 2022
dbd54d8
code cleanuo
Jan 10, 2022
71a5604
Module doc str
Jan 10, 2022
634823c
consistent arg order of NewFunction
Jan 10, 2022
6a0c11e
license and formatting
Jan 10, 2022
6242836
comment and cruft cleanup
Jan 10, 2022
61c7830
renamed Store to ModuleStore + docs
Jan 10, 2022
bb98aa8
added import docs
Jan 10, 2022
36b1018
shadowing cleanup
Jan 10, 2022
f2602bc
docs tweaks
Jan 10, 2022
e25e3bb
code cleanup
Jan 10, 2022
9421b6b
synced with gpython
Jan 10, 2022
92aed8a
Merge branch 'go-python:master' into master
drew-512 Jan 10, 2022
96515ea
Update modules/runtime.go
drew-512 Jan 10, 2022
f62f2b7
Update modules/runtime.go
drew-512 Jan 10, 2022
3bdba62
reverted from %w to %v (linux CI compile failure)
Jan 10, 2022
8033615
renamed py.Ctx to py.Context
Jan 10, 2022
681abf2
added Context.Close() and ModuleImpl.OnContextClosed()
Jan 18, 2022
7508556
docs and LoadIntsFromList
Jan 19, 2022
99bcb40
rename edits
Jan 21, 2022
0cc7650
push/popBusy now private
Jan 21, 2022
9afb2d3
doc edits
Jan 21, 2022
7d2a8d9
fixed kwarg issues in ParseTupleAndKeywords() and builtin_print test
Jan 25, 2022
ba7db80
added Flush kwarg to print test
Jan 25, 2022
61e42c8
added embedding example
Jan 26, 2022
168c337
main README makeover
Jan 26, 2022
2b7e1ab
fixed type conversion
Jan 26, 2022
3821b4b
Update README.md
drew-512 Jan 28, 2022
170f0d2
Update README.md
drew-512 Jan 28, 2022
b68b9f6
Update py/util.go
drew-512 Jan 28, 2022
759023b
Update py/util.go
drew-512 Jan 28, 2022
0c77716
Update py/util.go
drew-512 Jan 28, 2022
505755d
LoadIntsFromList cleanup
Jan 28, 2022
02d6b3e
comment edits
Jan 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
renamed py.Ctx to py.Context
  • Loading branch information
Drew O'Meara committed Jan 10, 2022
commit 8033615317b96cab0fe92cb15a2c394d09f047cf
6 changes: 3 additions & 3 deletions builtin/builtin.go
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ func init() {
Methods: methods,
Globals: globals,
})

}

const print_doc = `print(value, ..., sep=' ', end='\\n', file=sys.stdout, flush=False)
Expand All @@ -190,7 +190,7 @@ func builtin_print(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Obje
endObj py.Object = py.String("\n")
flush py.Object
)
sysModule, err := self.(*py.Module).Ctx.GetModule("sys")
sysModule, err := self.(*py.Module).Context.GetModule("sys")
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -463,7 +463,7 @@ func builtin___build_class__(self py.Object, args py.Tuple, kwargs py.StringDict
}
// fmt.Printf("Calling %v with %v and %v\n", fn.Name, fn.Globals, ns)
// fmt.Printf("Code = %#v\n", fn.Code)
cell, err = fn.Ctx.RunCode(fn.Code, fn.Globals, ns, fn.Closure)
cell, err = fn.Context.RunCode(fn.Code, fn.Globals, ns, fn.Closure)
if err != nil {
return nil, err
}
Expand Down
10 changes: 5 additions & 5 deletions examples/multi-ctx/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import (
func main() {

// The total job count implies a fixed amount of work.
// The number of workers is how many py.Ctx (in concurrent goroutines) to pull jobs off the queue.
// The number of workers is how many py.Context (in concurrent goroutines) to pull jobs off the queue.
// One worker does all the work serially while N number of workers will (ideally) divides up.
totalJobs := 20

Expand Down Expand Up @@ -55,7 +55,7 @@ print("%s ready!" % (WORKER_ID))

type worker struct {
name string
ctx py.Ctx
ctx py.Context
main *py.Module
job *py.Code
}
Expand Down Expand Up @@ -96,14 +96,14 @@ func RunMultiPi(numWorkers, numTimes int) time.Duration {
workers := make([]worker, numWorkers)
for i := 0; i < numWorkers; i++ {

opts := py.DefaultCtxOpts()
opts := py.DefaultContextOpts()

// Make sure our import statement will find pi_chudnovsky_bs
opts.SysPaths = append(opts.SysPaths, "..")

workers[i] = worker{
name: fmt.Sprintf("Worker #%d", i+1),
ctx: py.NewCtx(opts),
ctx: py.NewContext(opts),
job: jobCode,
}

Expand All @@ -116,7 +116,7 @@ func RunMultiPi(numWorkers, numTimes int) time.Duration {
w := workers[i]
go func() {

// Compiling can be concurrent since there is no associated py.Ctx
// Compiling can be concurrent since there is no associated py.Context
w.compileTemplate(jobSrcTemplate)

for jobID := range jobPipe {
Expand Down
12 changes: 6 additions & 6 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@ func main() {
flag.Usage = syntaxError
flag.Parse()
args := flag.Args()
opts := py.DefaultCtxOpts()

opts := py.DefaultContextOpts()
opts.SysArgs = flag.Args()
ctx := py.NewCtx(opts)
ctx := py.NewContext(opts)

if *cpuprofile != "" {
f, err := os.Create(*cpuprofile)
if err != nil {
Expand All @@ -59,7 +59,7 @@ func main() {
}
defer pprof.StopCPUProfile()
}

// IF no args, enter REPL mode
if len(args) == 0 {

Expand All @@ -70,7 +70,7 @@ func main() {

replCtx := repl.New(ctx)
cli.RunREPL(replCtx)

} else {
_, err := py.RunFile(ctx, args[0], py.CompileOpts{}, nil)
if err != nil {
Expand Down
26 changes: 13 additions & 13 deletions modules/runtime.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,19 @@ import (
)

func init() {
// Assign the base-level py.Ctx creation function while also preventing an import cycle.
py.NewCtx = NewCtx
// Assign the base-level py.Context creation function while also preventing an import cycle.
py.NewContext = NewContext
}

// ctx implements py.Ctx
type ctx struct {
// context implements py.Context
type context struct {
store *py.ModuleStore
opts py.CtxOpts
opts py.ContextOpts
}

// See py.Ctx interface
func NewCtx(opts py.CtxOpts) py.Ctx {
ctx := &ctx{
// See py.Context interface
func NewContext(opts py.ContextOpts) py.Context {
ctx := &context{
opts: opts,
}

Expand All @@ -50,7 +50,7 @@ func NewCtx(opts py.CtxOpts) py.Ctx {
return ctx
}

func (ctx *ctx) ModuleInit(impl *py.ModuleImpl) (*py.Module, error) {
func (ctx *context) ModuleInit(impl *py.ModuleImpl) (*py.Module, error) {
var err error

if impl.Code == nil && len(impl.CodeSrc) > 0 {
Expand Down Expand Up @@ -87,7 +87,7 @@ func (ctx *ctx) ModuleInit(impl *py.ModuleImpl) (*py.Module, error) {
return module, nil
}

func (ctx *ctx) ResolveAndCompile(pathname string, opts py.CompileOpts) (py.CompileOut, error) {
func (ctx *context) ResolveAndCompile(pathname string, opts py.CompileOpts) (py.CompileOut, error) {
tryPaths := defaultPaths
if opts.UseSysPaths {
tryPaths = ctx.Store().MustGetModule("sys").Globals["path"].(*py.List).Items
Expand Down Expand Up @@ -215,14 +215,14 @@ func resolveRunPath(runPath string, opts py.CompileOpts, pathObjs []py.Object, t
return err
}

func (ctx *ctx) RunCode(code *py.Code, globals, locals py.StringDict, closure py.Tuple) (py.Object, error) {
func (ctx *context) RunCode(code *py.Code, globals, locals py.StringDict, closure py.Tuple) (py.Object, error) {
return vm.EvalCode(ctx, code, globals, locals, nil, nil, nil, nil, closure)
}

func (ctx *ctx) GetModule(moduleName string) (*py.Module, error) {
func (ctx *context) GetModule(moduleName string) (*py.Module, error) {
return ctx.store.GetModule(moduleName)
}

func (ctx *ctx) Store() *py.ModuleStore {
func (ctx *context) Store() *py.ModuleStore {
return ctx.store
}
6 changes: 3 additions & 3 deletions py/frame.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type TryBlock struct {
// A python Frame object
type Frame struct {
// Back *Frame // previous frame, or nil
Ctx Ctx // host module (state) context
Context Context // host module (state) context
Code *Code // code segment
Builtins StringDict // builtin symbol table
Globals StringDict // global symbol table
Expand Down Expand Up @@ -78,7 +78,7 @@ func (o *Frame) Type() *Type {
}

// Make a new frame for a code object
func NewFrame(ctx Ctx, globals, locals StringDict, code *Code, closure Tuple) *Frame {
func NewFrame(ctx Context, globals, locals StringDict, code *Code, closure Tuple) *Frame {
nlocals := int(code.Nlocals)
ncells := len(code.Cellvars)
nfrees := len(code.Freevars)
Expand All @@ -91,7 +91,7 @@ func NewFrame(ctx Ctx, globals, locals StringDict, code *Code, closure Tuple) *F
cellAndFreeVars := allocation[nlocals:varsize]

return &Frame{
Ctx: ctx,
Context: ctx,
Globals: globals,
Locals: locals,
Code: code,
Expand Down
8 changes: 4 additions & 4 deletions py/function.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ package py
// A python Function object
type Function struct {
Code *Code // A code object, the __code__ attribute
Ctx Ctx // Host VM context
Context Context // Host VM context
Globals StringDict // A dictionary (other mappings won't do)
Defaults Tuple // NULL or a tuple
KwDefaults StringDict // NULL or a dict
Expand Down Expand Up @@ -56,7 +56,7 @@ func (f *Function) GetDict() StringDict {
// attribute. qualname should be a unicode object or ""; if "", the
// __qualname__ attribute is set to the same value as its __name__
// attribute.
func NewFunction(ctx Ctx, code *Code, globals StringDict, qualname string) *Function {
func NewFunction(ctx Context, code *Code, globals StringDict, qualname string) *Function {
var doc Object
if len(code.Consts) >= 1 {
doc = code.Consts[0]
Expand All @@ -73,7 +73,7 @@ func NewFunction(ctx Ctx, code *Code, globals StringDict, qualname string) *Func

return &Function{
Code: code,
Ctx: ctx,
Context: ctx,
Qualname: qualname,
Globals: globals,
Name: code.Name,
Expand All @@ -84,7 +84,7 @@ func NewFunction(ctx Ctx, code *Code, globals StringDict, qualname string) *Func

// Call a function
func (f *Function) M__call__(args Tuple, kwargs StringDict) (Object, error) {
result, err := VmEvalCode(f.Ctx, f.Code, f.Globals, NewStringDict(), args, kwargs, f.Defaults, f.KwDefaults, f.Closure)
result, err := VmEvalCode(f.Context, f.Code, f.Globals, NewStringDict(), args, kwargs, f.Defaults, f.KwDefaults, f.Closure)
if err != nil {
return nil, err
}
Expand Down
8 changes: 4 additions & 4 deletions py/import.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
"strings"
)

func Import(ctx Ctx, names ...string) error {
func Import(ctx Context, names ...string) error {
for _, name := range names {
_, err := ImportModuleLevelObject(ctx, name, nil, nil, nil, 0)
if err != nil {
Expand Down Expand Up @@ -80,7 +80,7 @@ func Import(ctx Ctx, names ...string) error {
//
// Changed in version 3.3: Negative values for level are no longer
// supported (which also changes the default value to 0).
func ImportModuleLevelObject(ctx Ctx, name string, globals, locals StringDict, fromlist Tuple, level int) (Object, error) {
func ImportModuleLevelObject(ctx Context, name string, globals, locals StringDict, fromlist Tuple, level int) (Object, error) {
// Module already loaded - return that
if module, err := ctx.GetModule(name); err == nil {
return module, nil
Expand Down Expand Up @@ -124,7 +124,7 @@ func ImportModuleLevelObject(ctx Ctx, name string, globals, locals StringDict, f
// This calls functins from _bootstrap.py which is a frozen module
//
// Too much functionality for the moment
func XImportModuleLevelObject(ctx Ctx, nameObj, given_globals, locals, given_fromlist Object, level int) (Object, error) {
func XImportModuleLevelObject(ctx Context, nameObj, given_globals, locals, given_fromlist Object, level int) (Object, error) {
var abs_name string
var builtins_import Object
var final_mod Object
Expand Down Expand Up @@ -338,7 +338,7 @@ error:
}

// The actual import code
func BuiltinImport(ctx Ctx, self Object, args Tuple, kwargs StringDict, currentGlobal StringDict) (Object, error) {
func BuiltinImport(ctx Context, self Object, args Tuple, kwargs StringDict, currentGlobal StringDict) (Object, error) {
kwlist := []string{"name", "globals", "locals", "fromlist", "level"}
var name Object
var globals Object = currentGlobal
Expand Down
22 changes: 11 additions & 11 deletions py/module.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,23 @@ import (
type ModuleFlags int32

const (
// Set for modules that are threadsafe, stateless, and/or can be shared across multiple py.Ctx instances (for efficiency).
// Otherwise, a separate module instance is created for each py.Ctx that imports it.
// Set for modules that are threadsafe, stateless, and/or can be shared across multiple py.Context instances (for efficiency).
// Otherwise, a separate module instance is created for each py.Context that imports it.
ShareModule ModuleFlags = 0x01 // @@TODO

MainModuleName = "__main__"
)

// ModuleInfo contains info and about a module and can specify flags that affect how it is imported into a py.Ctx
// ModuleInfo contains info and about a module and can specify flags that affect how it is imported into a py.Context
type ModuleInfo struct {
Name string // __name__ (if nil, "__main__" is used)
Doc string // __doc__
FileDesc string // __file__
Flags ModuleFlags
}

// ModuleImpl is used for modules that are ready to be imported into a py.Ctx.
// If a module is threadsafe and stateless it can be shared across multiple py.Ctx instances (for efficiency).
// ModuleImpl is used for modules that are ready to be imported into a py.Context.
// If a module is threadsafe and stateless it can be shared across multiple py.Context instances (for efficiency).
// By convention, .Code is executed when a module instance is initialized.
// If .Code == nil, then .CodeBuf or .CodeSrc will be auto-compiled to set .Code.
type ModuleImpl struct {
Expand All @@ -42,7 +42,7 @@ type ModuleImpl struct {
Code *Code // Module code body
}

// ModuleStore is a container of Module imported into an owning py.Ctx.
// ModuleStore is a container of Module imported into an owning py.Context.
type ModuleStore struct {
// Registry of installed modules
modules map[string]*Module
Expand Down Expand Up @@ -85,12 +85,12 @@ func NewModuleStore() *ModuleStore {
}
}

// Module is a runtime instance of a ModuleImpl bound to the py.Ctx that imported it.
// Module is a runtime instance of a ModuleImpl bound to the py.Context that imported it.
type Module struct {
ModuleInfo

Globals StringDict
Ctx Ctx
Context Context
}

var ModuleType = NewType("module", "module object")
Expand All @@ -112,17 +112,17 @@ func (m *Module) GetDict() StringDict {
// NewModule adds a new Module instance to this ModuleStore.
// Each given Method prototype is used to create a new "live" Method bound this the newly created Module.
// This func also sets appropriate module global attribs based on the given ModuleInfo (e.g. __name__).
func (store *ModuleStore) NewModule(ctx Ctx, info ModuleInfo, methods []*Method, globals StringDict) (*Module, error) {
func (store *ModuleStore) NewModule(ctx Context, info ModuleInfo, methods []*Method, globals StringDict) (*Module, error) {
if info.Name == "" {
info.Name = MainModuleName
}
m := &Module{
ModuleInfo: info,
Globals: globals.Copy(),
Ctx: ctx,
Context: ctx,
}
// Insert the methods into the module dictionary
// Copy each method an insert each "live" with a ptr back to the module (which can also lead us to the host Ctx)
// Copy each method an insert each "live" with a ptr back to the module (which can also lead us to the host Context)
for _, method := range methods {
methodInst := new(Method)
*methodInst = *method
Expand Down
2 changes: 1 addition & 1 deletion py/py.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type IGoInt64 interface {

var (
// Set in vm/eval.go - to avoid circular import
VmEvalCode func(ctx Ctx, code *Code, globals, locals StringDict, args []Object, kws StringDict, defs []Object, kwdefs StringDict, closure Tuple) (retval Object, err error)
VmEvalCode func(ctx Context, code *Code, globals, locals StringDict, args []Object, kws StringDict, defs []Object, kwdefs StringDict, closure Tuple) (retval Object, err error)
VmRunFrame func(frame *Frame) (res Object, err error)
)

Expand Down
Loading