• MDP Generator through Prolog and LLMs
  • Index
  • Compatibility
  • Prerequisites
    • Python3
    • Large Language Models
      • Azure OpenAI GPT
      • HuggingFace
  • Installing
    • Script install.sh
    • Docker
  • Running
    • Running using the command line
      • Generating MDP from Prolog KB
      • Generating the MDP policy through Storm
      • Generating the MDP policy with the Python3 script

The main problem of running the code from source with MacOS is the missing compatibility of Stormpy.

The script install.sh will take care of installing the required packages (also the ones for python3, see below).

• SWI-Prolog
• Storm model checker

If you want to use the solver wrapper in python3 inside storm_solver, then the following packages are also required:

• Stormpy, which in turn requires:
  • pycarl, which in turn requires:
    • carl-storm

Some of these packages can be installed through PPA (Ubuntu) or Brew (MacOS), but we recommend installing them from source as in the script install.sh.

We found that stormpy needs to be patched to be run correctly with the simulato, see install/simulator.patch.

The following dependencies, minus the API keys, are automatically installed when installing from the LLM/requirements.txt file. If you are using Docker, you can skip this step.

If you want to use one model in particular, we suggest to run the correct requirements file from the directory you are interested in.

If using Azure OpenAI GPT, you first need to obtain an API key. You can do this by going to the Azure webpage and follow the instructions there. Once you have the API information, you should create/update a config file within LLM/LLMAzureOpenAI/config. The API key shall be placed inside the LLM/LLMAzureOpenAI/.env file, which will then be read automatically by the script to get the key. The file should look like this:

As for the required libraries, you can install them using the following command:

pip install -r LLM/LLMAzureOpenAI/requirements.txt

NOT IMPLEMENTED YET

As said, one can choose to install the dependency on their-own, use the install.sh script, or use Docker.

Simply run:

chmod +x install.sh && ./install.sh

This will:

• install the dependencies need to compile the packages mentioned in Prerequisites;
• download the GitHub repos for storm, carl-storm, pycarl, stormpy;
• compile storm (and add it to the environment variables), carl-storm, pycarl and stormpy.

Use -h to see the options for the script:

Usage: install.sh [options]
Options:
  -h            Show this help message and exit
  -V            Enable creating a virtual environment (default: disabled)
  -j <jobs>     Specify the number of parallel jobs for building (default: 1)
  -l            Install dependencies for LLM
  -p            Install dependencies for Prolog
  -P            Install dependencies for Prism
  -s            Install dependencies for Storm
  -v            Run with commands output. Default is not enabled
By default, the script installs all the dependencies (Swipl, Storm, Prism and the LLM)

Two Docker files are provided:

• Dockerfile.storm: uses the default image of Storm: movesrwth/storm.
• Dockerfile.ubuntu: is based on an image of Ubuntu:22.04 and installs all the dependencies using install.sh.

Create the Docker images by running the following commands from the root of the framework folder:

docker build -t <image_name> -f Dockerfile.storm .

or

docker build -t <image_name> -f Dockerfile.ubuntu .

Both Docker files simply provide an environment in which to run the code, they do not have an entrypoint that will be executed automatically when running the container. Also, neither of them copies the code to the container. From within the framework folder, run them with:

docker run -it -v $(pwd):/app --rm <image_name> /bin/bash

Once the KB file has been generated and the include.pl file has been updated accordingly with the correct path to the KB file, then one can generate the Prism file by running the following command:

swipl -l mdp_generator/mdp_generator.pl -t 'mdp_generator.'

The Prolog script will generate the Prism file storm_solve/mdp.pm, which can then be used by Storm to find the best policy.

# To generate the policy based on the maximum probability of reaching a final state
storm --prism mdp.pm --prop 'Pmax=? [F "doneP"]' --minmax:method vi --exportscheduler highP.sched --buildstateval --buildchoiceorig --buildfull

# To generate the best policy based on the reward of the MDP transitions
storm --prism mdp.pm --prop 'Rmax=? [F "doneR"]' --minmax:method vi --exportscheduler highR.sched --buildstateval --buildchoiceorig --buildfull

Otherwise, one can use the provided Python3 script:

python3 storm_solver/policy_extractor.py --help

This comes with a number of different options to choose and set:

usage: policy_extractor.py [-h] [-r] [-F FORMULA] [-P PRISM_FILE] [-O OUTPUT_DIR] [-o OUTPUT] [-s] [-S SEED] [-f] [-N N_PATHS] [-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--disable-console] [--force]

Extract optimal policy for MDP from .pm file using stormpy

options:
  -h, --help            show this help message and exit
  -r, --use-reward      Use reward-based optimization (default: probability-based, i.e., False)
  -F FORMULA, --formula FORMULA
                        Property formula to check on the MDP model. Overwrites --use-reward.
  -P PRISM_FILE, --prism-file PRISM_FILE
                        Path to the MDP model file (default: output/mdp_gen.pm)
  -O OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Directory where to save all the output (default: output)
  -o OUTPUT, --output OUTPUT
                        Output file for the state-action mapping (default: policy.py). If a path is passed, it will overwrite --output-dir.
  -s, --simulate        Simulate the MDP model (default: False)
  -S SEED, --seed SEED  Seed for random number generation during simulation. Default is 0. Set to 0 to use current time. Only used if --simulate is set.
  -f, --simulate-faulty
                        Simulate the MDP model with faulty states (default: False). Only used if --simulate is set.
  -N N_PATHS, --n-paths N_PATHS
                        The number of paths to simulate. Default: False. Only used if --simulate is set.
  -L {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level (default: DEBUG)
  --disable-console     Disable logger output to console
  --force               Force overwrite of the output directory if it already exists and is not empty

For example, we can use the following options to run the experiment 1 of the AGV scenario asking for both a solution and a simulation (which will generate also the policy.py file containing the state-action table) and saving all the output files in output:

python3 storm_solver/policy_extractor.py --log-level INFO --n-paths 10--simulate --prism-file exps/agv/1/GPT4/mdp-gen/mdp_gen.pm --output-dir output            

If -r is in the arguments list, the script will minimize the reward value.