Constrained Hierarchical Deep Reinforcement Learning with Differentiable Formal Specifications

Demos

4 robots perform different tasks below. Full demos are collected in a website.

drone-loop.mp4	point-seq.mp4
car-branch.mp4	doggo-signal.mp4

Install

1. Install this package and dependencies (e.g., pybullet, pytorch, etc.)

pip instal -e .

2. Follow this document to configure and troubleshoot MuJoCo(-py).

Examples

Differentiable Specifications

We implemented a simple LL parser for our differentiable TLTL specifications.

1. Syntax Examples: how to write TLTL specifications with pure python built-in operators.
2. Quantitative Semantics Examples: forward and backward through TLTL specifications.
3. Specifications: 5 types of specifications and solving them with gradient.

Goal-Conditioned Environments and Policies

Reach Random Goals: An example shows how the goal-conditioned environments and policies work.

Pretrained Policies

Run Pretrained Policies: Run our pretrained policies with

python examples/hrl/pretrained.py \
  --robot_name [point, car, doggo, drone] \
  --task_name [seq, cover, branch, loop, signal] \
  --gui \
  --n_eps [number_of_epochs]

for example, below command runs pretrained policy for point robot's seq task 5 times with gui

python examples/hrl/pretrained.py \
  --robot_name point \
  --task_name seq \
  --gui \
  --n_eps 5

Reuse and Citation

This code repository is packed as a standard python package. Install it with pip install -e ., and one will be able to revoke all the modules. Detailed examples for these modules are provided in examples/.

Tweaking configuration here can change the log level, PyTorch device, etc.

One related paper to this repository is under a double-blind review process. The BibTeX will be available after we can make it public.