This (early) end-of-year big release comes with a big 🎁: distributed execution. For anyone needing to scale the training (or inference), this is a must. As it goes with these new things, it should be considered experimental. I'll be actually using it, and hopefully any missing issues (including adding a distributed tutorial -- there is a demo already) will be solved by the end of the year, or early next year. 🦌

There is also some API clean ups -- mostly returning error in places that were panicking otherwise. So ... a bit of a "Grinch" gift, is the need for some updates, if you update to this version. All very trivial (added an error to the return list of some functions and methods).

We also had some external collaborations 🎉 Looking forward to more of those, there is so much interesting stuff to be done! (Quantization, using SIMD operations in the "Simple Go" backend, etc).

Hightlights:

• Distributed (cross-devices) execution: with AutoSharding and SPMD strategies;
  Also added support for "portable device" execution.

• API changes: (will require simple fixes)

  • Most not graph building APIs now return errors (as opposed to panicking). Graph building functions
    still use panic to return error -- otherwise it's too painful to express math.
  • All "Rng" renamed to "RNG" -- acronyms in Go are usually capitalized.

Distributed computation improvements and refactorings:

• Package graph:
  • Fixed/improved documentation.
  • Added IsNegative, IsPositive, IsNonNegative, IsNonPositive.
  • Added SubScalar and tests for the '*Scalar' functions.
  • Added Graph.WithDistributedStrategy, Graph.WithDeviceMesh. Graph.DeviceMesh and Graph.NumDevices
  • Added Graph.Distributed() with "collective" (across devices) operations (like AllReduce).
  • Renamed: s/Exec.InDevice/Exec.WithDevice; s/Exec.SetName/Exec.WithName
  • Added RunOnDevice.
  • Added Exec.AutoSharding and Exec.SPMD.
• Package context:
  • Added context.MustGetParam[T](ctx, key) and context.MustGetGraphParam[T](ctx, graph, key).
  • Added Exec.AutoSharding and Exec.SPMD.
  • Added Variable.DistributedValue and Variable.SetDistributedValue.
• Package train:
  • Added train.DistributedDataset and train.BaseDataset.
  • Dataset.Reset now returns an error.
  • Trainer.TrainStep, Trainer.EvalStep and Trainer.Eval now return errors as opposed to panicking.
  • Added Trainer.WithDeviceAssignment.
  • Added Trainer.DistributedTrainStep, Trainer.DistributedEvalStep and Trainer.DistributedEval.
• Package datasets:
  • Added datasets.DistributedAccumulator: converts a normal Dataset into a DistributedDataset.
  • Added datasets.OnDevice: pre-uploads data to devices.
• Package backend:
  • Added Backend.CopyToDevice
  • Builder.Parameter() now takes an optional ShardingSpec for sharded inputs.
  • Added ops: AllReduce
  • Backend.NumDevices() returns an int now.
  • Package backends/notimplemented:
    • Added dummy Backend that can be used to easily mock backends.
• Package pkg/core/distributed:
  -Added DeviceMesh, ShardSpec and distributed.Tensor objects.
• Package pkg/core/tensors:
  • Added Tensor.CheckValid(), Tensor.Device(), Tensor.Backend()
  • Changing it to return an error (as opposed to panic) where possible.

Other improvements:

• Package simplego:
  • Cleanups and improvements: thanks to @wunderbarb!
  • Fixed the issue with not handling the default value for the donate parameter in the Execute method.
• Package cosineschedule:
  • Added WarmUpSteps and NumCycles hyperparameters -- removed overloading of periodSteps.
• Added sponsorship badge and section to README.md. Also added the FUNDING.yml pointing to sponsorship.
• Added .golangci.yml and fixed many (still a long way to go) lint-warnings.
  • Based on https://gist.github.com/maratori/47a4d00457a92aa426dbd48a18776322
• GitHub actions (workflows):
  • Renamed tests to "Linux" and "Darwin."
  • Updated badges in README.md.
• Updated dependency to Gopjrt v0.8.5, fixing xlabuilder for new C compilers.
• Removed ui/fyneui:
  • It was incomplete, and it would be better offered as a separate package to avoid the dependencies.
• Package graph:
  • Added a negative and out-of-bounds indices test for Gather.
• Package simplego:
  • Partially fixed a race condition where the executable is finalized during the execution, causing crashes --
    Thanks @ajroetker!