sharing steps across studies #452
Description
It would be great to be able to share some steps across studies, e.g. some pre-processing
example
study:
- name: step1
description: preproc
run:
cmd: |
echo "Some preprocesisng code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
- name: step2
description: some code
run:
cmd: |
echo "Actual work"
$(LAUNCHER) $(CODE) -in $(MODEL1)
procs: 1
nodes: 1
walltime: "00:01:00"assuming step1 is shared by many maeastro studies it could become something like this:
study:
- name: step1
path: $(SHARED_MAESTRO_FILES)/preprocessing.yml
step: standard_preproc
- name: step2
description: some code
run:
cmd: |
echo "Actual work"
$(LAUNCHER) $(CODE) -in $(MODEL1)
procs: 1
nodes: 1
walltime: "00:01:00"
depends: step1And the preprocessing.yml file could look like this
study:
- name: standard_preproc
description: standard preprocessing
run:
cmd: |
echo "Some preprocessing code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
- name: advanced_preproc
description: a more advanced preprocessing step
run:
cmd: |
echo "Some preprocessing code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
That way we can share and update the preprocessing step across many studies. I do realize this is not great for provenance, but maestro could build a "Final" yml file with the pieces brought over from all the external yml files.
The concept could also be exported to resources where a resources.yml could exit in a standard place but with diffierent values depending on the machine you're running on.