This release starts rolling out changes to RL environments, along with quality-of-life improvements to the startup and build processes. One of them: agents can now retry connecting to the spine several times at startup, getting us rid of clunky timeouts 😉

RL changes and migration notes

RL environments now work with $R(o, a)$ reward functions. The refactoring adds an intermediate abstract class for environments that control Upkie in the wheeled inverted pendulum, and additional acceleration limits to the UpkieGroundVelocity environment (formerly UpkieWheelsEnv).

The refactoring also introduces a regulate_frequency boolean argument: the proper way not to regulate frequency is now regulate_frequency=False rather than frequency=None.

Enjoy these changes, and chime in in the Discussions if you have feedback 😃

Migration notes

• Instead of frequency=None, set regulate_frequency=False to disable loop frequency regulation (e.g. for training)
• Rename UpkieServosEnv to UpkieServos
• Rename UpkieWheelsEnv to UpkieGroundVelocity

Changelog

Added

• Bazel: Python library target for upkie module
• PPO balancer: Save policy configuration to a YAML file
• PPO balancer: Train with multiprocessing
• envs: Accessor to env.rate for logging purposes
• envs: Randomize base orientation and position on resets
• envs: InitRandomization dataclass to describe initial state randomization
• envs: UpkieGroundVelocity can include a velocity low-pass filter
• envs: UpkieGroundVelocity can limit ground acceleration as well
• envs: UpkieGroundVelocity low-pass filter can also be randomized
• utils: Log path utility functions

Changed

• Breaking: Environment rewards depend on both observation and action
• Breaking: Rename UpkieServosEnv to UpkieServos
• Breaking: Rename UpkieWheelsEnv to UpkieGroundVelocity
• Breaking: Use regulate_frequency env kwarg instead of frequency=None
• Makefile: Default wheel balancer config to the output of hostname
• Makefile: Rename ROBOT environment variable to UPKIE_NAME
• PPO balancer: Change training directory to /tmp/ppo_balancer
• PPO balancer: Policy CLI argument becomes positional and optional
• PPO balancer: Refactor agent settings
• PPO balancer: Rename effective_time_horizon to discounted_horizon_duration
• agents: Retry connecting to the spine several times at startup
• envs: Retry connecting to the spine several times at startup

Fixed

• Wheel balancer: Configure spine properly
• Wheel odometry: Check that observer is configured properly

Removed

• envs: Remove get_range from rewards as it is deprecated from Gymnasium

This release brings a great deal of fixes and build system improvements. Notably, we can now run agents both from Python or Bazel interchangeably. There was also a refactoring to the Reward class in environments: agents can now define their own task-specific rewards. The standing reward is now part of the PPO balancer, while at the environment level the default is simply a survival reward (+1 at each non-failing step).

Thanks to @boragokbakan for contributing to this release 👍

Added

• Makefile: Add clean rule to remove all intermediate files
• Makefile: Add coverage rule to check a local HTML report
• agents: Add --show CLI argument to the wheel balancer's Bullet target
• envs: Base Reward class for rewards
• envs: Survival reward, which is simply always one
• utils: Configure agent process on the Raspberry Pi

Changed

• MPC balancer: Can now be run both via Bazel or Python
• Makefile: Remove most agent targets to promote running their main.py
• PPO balancer: Can now be run both via Bazel or Python
• Pink balancer: Can now be run both via Bazel or Python
• Pink balancer: Remove unit tests
• README: Recommended way to run agents is now via Python
• Wheel balancer: Can now be run both via Bazel or Python
• envs: Default reward for all environments is now the survival reward
• envs: Move StandingReward to the PPO balancer
• examples: Remove CPU isolation example, now a utils.raspi function call
• tools: CPU scaling scripts don't need to be run as root any more
• utils: Remove realtime submodule in favor of raspi

Fixed

• PPO balancer: Configure main script process on the Raspberry Pi
• PPO balancer: Disable rate limiter during training
• Wheel balancer: Handle SpineInterface failures when forking a Bullet simulation (thanks to @boragokbakan)
• observers: Check whether floor contact observer is initialized properly
• tools: Fix permissions of vcgencheck

This patch release fixes environment registration and dependencies in the upkie PyPI package.

Added

• envs: Inform user on installing all optional dependencies
• envs: Name environment rate limiters for more readable logging

Changed

• envs: Refactor environment registration function

Fixed

• PyPI: add PyYAML to dependencies as it is needed by upkie.config
• envs: Export register from submodule
• envs: Overlay constructor spine configuration on top of default config

With this patch release, we skip environment registration when a dependency is missing. This change allows environments with optional dependencies and benefits UpkieServosEnv.

Changed

• Bump scipy dependency from 1.8.0 to 1.10.0
• envs: Skip environment registration upon missing dependency

This patch release actually fixes the missing distribution of upkie.config on PyPI, and adds a corresponding test to catch such errors in continuous integration.

Fixed

• config: Distribute missing config submodule in PyPI package

This patch release fixes the missing distribution of upkie.config on PyPI.

Fixed

• config: Add missing config submodule to PyPI
• Fix source code distribution of PyPI package

This release adds support for developing on macOS, contributed by @boragokbakan. It also introduces a top-level config submodule for general robot configuration, and the usual harvest of bug fixes and small improvements.

Added

• Support macOS operating systems, thanks to @boragokbakan 👍
• config: Top-level configuration submodule for robot-wide configuration
• spines: Build and deploy the mock spine to the Raspberry Pi
• tools: CPU frequency scaling scripts
• utils: Function to detect when we run on a Raspberry Pi

Changed

• deps: Update loop-rate-limiters to 0.5.0
• envs: Rename config parameter to a more explicit spine_config

Fixed

• PPO balancer: fix time-limit import

This release fixes the version number of UpkieWheelsEnv-v4 (which was overlooked in v1.2.0).

This release makes an important update to the UpkieWheelsEnv environment: vectorized observations are now ordered as

0. Base pitch angle
1. Ground position
2. Base (not IMU) angular velocity
3. Ground velocity

In case of doubt, refer to the documentation for the semantics of observations and actions.

Added

• tools: add vcgencheck utility script
• tools: make hard_rezero search for upkie_tool and skip if not found

Changed

• build: Compile in optimized mode by default (previsouly: fast build)
• envs: Observation vector reordered, angular velocity is now that of the base.

Fixed

• build: Only run lint tests when --config lint is supplied
• envs: Make sure vectorized observations are float32

This release improves quality of use for deploying agents to the robots, and fixes IMU orientation and servo IDs to a proper convention (documented in the build instructions) for all robots.

It also improves Gym environments for training new agents, thanks to @perrin-isir 👍

Added

• agents: Detect config from hostname when running on the Pi
• envs: Allow faster-than-realtime Gym environment execution
• envs: Getter for environment frequency

Changed

• agents: Rename "test balancer" to "wheel balancer"
• agents: Update Pink balancer to the latest version of the library
• Makefile: Check that ROBOT environment variable is defined
• tools: Automatically sudo when running upkie_tool

Fixed

• observers: Add missing observer configuration in base Upkie environment
• observers: Configure IMU orientation in the base pitch observer
• spines: Fix joint IDs to left leg (1, 2, 3), right leg (4, 5, 6)
• workspace: Update Vulp for IMU frame simulation fix
• workspace: Update upkie_description for IMU orientation fix