LLNL · jwhite242 · Apr 28, 2023 · Jun 20, 2022 · Jun 20, 2022 · Jun 20, 2022
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -9,3 +9,13 @@ build:
       - pip install poetry>=1.2
       - poetry config virtualenvs.create false
       - poetry install
+
+mkdocs:
+  # configuration: docs/mkdocs.yml
+  # Temporarily allow warnings
+  fail_on_warning: false
+
+# # Optionally declare the Python requirements required to build your docs
+# python:
+#    install:
+#    - requirements: docs/requirements.txt
diff --git a/docs/Doxyfile b/docs/Doxyfile
diff --git a/docs/LICENSE.md b/docs/LICENSE.md
@@ -0,0 +1,21 @@
+### The MIT License (MIT)
+
+Copyright (c) 2021 Francesco Di Natale
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
diff --git a/docs/Maestro/basics.md b/docs/Maestro/basics.md
@@ -0,0 +1,116 @@
+![](https://github.com/LLNL/maestrowf/raw/develop/assets/logo.png?raw=true "Orchestrate your workflows with ease!")
+
+# Maestro Workflow Conductor ([maestrowf](https://pypi.org/project/maestrowf/))
+
+[![Build Status](https://travis-ci.org/LLNL/maestrowf.svg?branch=develop)](https://travis-ci.org/LLNL/maestrowf)
+[![PyPI](https://img.shields.io/pypi/v/maestrowf.svg)](https://pypi.python.org/pypi?name=maestrowf&version=1.0.0&:action=display)
+![Spack](https://img.shields.io/spack/v/py-maestrowf)
+[![Issues](https://img.shields.io/github/issues/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/issues)
+[![Forks](https://img.shields.io/github/forks/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/network)
+[![Stars](https://img.shields.io/github/stars/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/stargazers)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/LLNL/maestrowf/master/LICENSE)
+
+[![Downloads](https://static.pepy.tech/personalized-badge/maestrowf?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/maestrowf)
+[![Downloads](https://static.pepy.tech/personalized-badge/maestrowf?period=month&units=international_system&left_color=grey&right_color=blue&left_text=Downloads/Month)](https://pepy.tech/project/maestrowf)
+
+Maestro can be installed via [pip](https://pip.pypa.io/):
+
+    pip install maestrowf
+
+## Documentation
+
+* [Maestro Documentation](https://maestrowf.readthedocs.io) - Official Maestro documentation.
+* [Maestro Sheetmusic](https://github.com/LLNL/maestro_sheetmusic) - A collection of sample and user contributed Maestro study examples.
+* [Maestro Samples](/samples) - Maestro sample studies.
+
+## Getting Started is Quick and Easy
+
+Create a `YAML` file named `study.yaml` and paste the following content into the file:
+
+``` yaml
+description:
+    name: hello_world
+    description: A simple 'Hello World' study.
+
+study:
+    - name: say-hello
+      description: Say hello to the world!
+      run:
+          cmd: |
+            echo "Hello, World!" > hello_world.txt
+```
+
+> *PHILOSOPHY*: Maestro believes in the principle of a clearly defined process, specified as a list of tasks, that are self-documenting and clear in their intent.
+
+Running the `hello_world` study is as simple as...
+
+    maestro run study.yaml
+
+## Creating a Parameter Study is just as Easy
+
+With the addition of the `global.parameters` block, and a few simple tweaks to your `study` block, the complete specification should look like this:
+
+``` yaml
+description:
+    name: hello_planet
+    description: A simple study to say hello to planets (and Pluto)
+
+study:
+    - name: say-hello
+      description: Say hello to a planet!
+      run:
+          cmd: |
+            echo "Hello, $(PLANET)!" > hello_$(PLANET).txt
+
+global.parameters:
+    PLANET:
+        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto]
+        label: PLANET.%%
+```
+
+> *PHILOSOPHY*: Maestro believes that a workflow should be easily parameterized with minimal modifications to the core process.
+
+Maestro will automatically expand each parameter into its own isolated workspace, generate a script for each parameter, and automatically monitor execution of each task.
+
+And, running the study is still as simple as:
+
+``` bash
+    maestro run study.yaml
+```
+
+## Scheduling Made Simple
+
+But wait there's more! If you want to schedule a study, it's just as simple. With some minor modifications, you are able to run on an [HPC](https://en.wikipedia.org/wiki/Supercomputer) system.
+
+``` yaml
+description:
+    name: hello_planet
+    description: A simple study to say hello to planets (and Pluto)
+
+batch:
+    type:  slurm
+    queue: pbatch
+    host:  quartz
+    bank:  science
+
+study:
+    - name: say-hello
+      description: Say hello to a planet!
+      run:
+          cmd: |
+            echo "Hello, $(PLANET)!" > hello_$(PLANET).txt
+          nodes: 1
+          procs: 1
+          walltime: "00:02:00"
+
+global.parameters:
+    PLANET:
+        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto]
+        label: PLANET.%%
+```
+
+> **NOTE**: This specification is configured to run on LLNL's quartz cluster. Under the `batch` header, you will need to make the necessary changes to schedule onto other HPC resources.
+>
+> *PHILOSOPHY*: Maestro believes that how a workflow is defined should be decoupled from how it's run. We achieve this capability by providing a seamless interface to multiple schedulers that allows Maestro to readily port workflows to multiple platforms.
+
+For other samples, see the [samples](/samples) subfolder. To continue with our Hello World example, see the [Basics of Study Construction](https://maestrowf.readthedocs.io/en/latest/hello_world.html) in our [documentation](https://maestrowf.readthedocs.io/en/latest/index.html).
diff --git a/docs/Maestro/cli.md b/docs/Maestro/cli.md
@@ -0,0 +1,129 @@
+# Command Line Interface
+---
+
+Installing Maestro will add two entry points/console scripts that are used for running, creating, and interacting with studies:
+
+* [`maestro`](#maestro)
+
+    This is the primary user interface for creating, launching, and monitoring studies.
+
+* [`conductor`](#conductor)
+
+    This is the background process that Maestro launches which does the actual orchestration and running of the studies and lives until the study execution is complete.
+
+
+## **maestro**
+
+The Maestro Workflow Conductor for specifying, launching, and managing general workflows.
+
+**Usage:**
+
+``` console
+maestro [OPTIONS] COMMAND [ARGS]...
+```
+
+**Options:**
+
+| Name | Type | Description | Default |
+| ---- | ---- | ----------- | ------- |
+| `-h`, `--help` | boolean | Show this help message and exit. | `False` |
+| `-l`, `--logpath` | filename | Alternate path to store program logging. | <study_name\>.log |
+| `-d`, `--debug_lvl` | choice (`1` &#x7C; `2` &#x7C; `3` &#x7C; `4` &#x7C; `5`) | Level of logging messages to be output.  Smaller choice produces more output. <table>  <thead>  <th>Choice</th>  <th>Max Debug Level</th>  </thead>  <tbody>  <tr>  <td>5</td>  <td>Critical</td>  </tr>  <tr>  <td>4</td>  <td>Error</td>  </tr>  <tr>  <td>3</td>  <td>Warning</td>  </tr>  <tr>  <td>2</td>  <td>Info (Default)</td>  </tr> <tr> <td>1</td> <td>Debug</td>  </tbody>  </table>               | 2 (Info) |
+| `-c`, `--logstdout` | boolean | Log to stdout in addition to a file.  **NOTE:** This only controls immediate output from `maestro` during the setup phase.  Once `conductor` is launched and the study is running this log info goes to file only if not running in the foreground | `True` | <!-- insert link to the foreground/background option -->
+| `-v`, `--version` | boolean | Show program's version number and exit. | `False` |
+
+
+**Subcommands**
+
+- [*cancel*](#cancel): Cancel all running jobs.
+- [*run*](#run): Launch a study based on a specification
+- [*status*](#status): Check the status of a running study.
+
+### **cancel**
+
+Cancel all running jobs in every given study directory.
+
+**Usage:**
+
+``` console
+maestro cancel [OPTIONS] DIRECTORY [DIRECTORY ...]
+```
+
+**Options:**
+
+| Name | Type | Description | Default |
+| ---- | ---- | ----------- | ------- |
+| `-h`, `--help` | boolean | Show this help message and exit. | `False` |
+
+
+### **run**
+
+Entry point for launching a study described in a YAML based study specification <!-- add link to this specification? -->
+
+**Usage:**
+
+``` console
+maestro run [OPTIONS] SPECIFICATION
+```
+
+**Options:**
+
+| Name | Type | Description | Default |
+| ---- | ---- | ----------- | ------- |
+| `-h`, `--help` | boolean | Show this help message and exit. | `False` |
+| `-a`, `--attempts` | integer | Maximum number of submission attempts before a step is marked failed. | 1 |
+| `-r`, `--rlimit` | integer | Maximum number of restarts allowed when steps specify a restart command. (0 denotes no limit)| 1 |
+| `-t`, `--throttle` | integer | Maximum number of inflight jobs allowed to execute simultaneously (0 denotes not throttling) | 0 |
+| `-s`, `--sleeptime` | integer | Amount of time (in seconds) for the manager to wait between job status checks. | 60 |
+| `--dry` | boolean | Generate the directory structure and scripts for a study but do not launch it. | `False` |
+| `-p`, `--pgen` | filename/path | Path to a Python code file containing a function that returns a custom filled ParameterGenerator instance. | None |
+| `--pargs` | string | A string that represents a single argument to pass a custom parameter generation function. Reuse '--parg' to pass multiple arguments. [Use with '--pgen'] | None |
+| `-o`, `--out` | path | Output path to place study in. [NOTE: overrides OUTPUT_PATH in the specified specification] | "<study name\>_timestamp" |
+| `-fg` | boolean | Runs the backend conductor in the foreground instead of using nohup. | `False` |
+| `--hashws` | boolean | Enable hashing of subdirectories in parameterized studies (NOTE: breaks commands that use parameter labels to search directories). | `False` |
+| `-n`, `--autono` | boolean | Automatically answer no to input prompts. | `False` |
+| `-y`, `--autoyes` | boolean | Automatically answer yes to input prompts. | `False` |
+| `--usetmp` | boolean | Make use of a temporary directory for dumping scripts and other Maestro related files. | `False` |
+
+
+### **status**
+
+Check the status of each given running or completed study.
+
+**Usage:**
+
+``` console
+maestro status [OPTIONS] DIRECTORY [DIRECTORY ...]
+```
+
+**Options:**
+
+| Name | Type | Description | Default |
+| ---- | ---- | ----------- | ------- |
+| `-h`, `--help` | boolean | Show this help message and exit. | `False` |
+| `--layout` | choice (`flat` &#x7C; `legacy` &#x7C; `narrow`) | Alternate status table layouts. See [Status Layouts](monitoring.md#status-layouts) for description of these options| `flat` |
+
+
+## **conductor**
+
+A application for checking and managing and ExecutionDAG within an executing study.
+
+**Usage:**
+
+``` console
+conductor [OPTIONS] DIRECTORY
+```
+
+**Options:**
+
+| Name | Type | Description | Default |
+| ---- | ---- | ----------- | ------- |
+| `-h`, `--help` | boolean | Show this help message and exit. | `False` |
+| `-s`, `--status` | path |  Check the status of the ExecutionGraph located as specified by the 'directory' argument. | None |
+| `-l`, `--logpath` | filename | Alternate path to store program logging. | <study_name\>.log |
+| `-d`, `--debug_lvl` | choice (`1` &#x7C; `2` &#x7C; `3` &#x7C; `4` &#x7C; `5`) | Level of logging messages to be output.  Smaller choice produces more output. <table>  <thead>  <th>Choice</th>  <th>Max Debug Level</th> </thead>  <tbody>  <tr>  <td>5</td>  <td>Critical</td>  </tr>  <tr>  <td>4</td>  <td>Error</td>  </tr>  <tr>  <td>3</td>  <td>Warning</td>  </tr>  <tr>  <td>2</td>  <td>Info (Default)</td>  </tr> <tr> <td>1</td> <td>Debug</td>  </tbody>  </table>               | 2 (Info) |
+| `-c`, `--logstdout` | boolean | Log to stdout in addition to a file.  **NOTE:** This only controls immediate output from `maestro` during the setup phase.  Once `conductor` is launched and the study is running this log info goes to file only if not running in the foreground | `True` | <!-- insert link to the foreground/background option -->
+| `-s`, `--sleeptime` | integer | Amount of time (in seconds) for the manager to wait between job status checks. | 60 |
+
+
+