Extra Flux Options #459

jwhite242 · 2025-08-15T00:19:32Z

Expands available options to pass through to flux, replacing the args dictionary in the batch block which only passed through shell options (--setopt/-o) to the launcher (flux run). Adds two dictionaries to the batch block:

allocation_args: enable attaching arbitrary numbers of options to the allocation (jobspec) corresponding to the cli equivalents: -o/--setopt, -S/--setattr, and --conf
launcher_args: replaces args for attaching to launcher, but enables passing through any option prefix, not just -o/--setopt

Deprecates args, but leaves it intact as the only option for older flux adapters (< 0.49) pending future removal.

Adds flux directives with all allocation options to serialized step scripts to enable reproducibility and direct cli submission by users to match capability of other script adapters.

Exposes new exclusive syntax for step keys to enable separate allocation/launcher control: only flux uses the launcher option, other schedulers to follow.

Adds script writing and job submission tests for both flux and slurm.

…llocation control

…andling in flux interfaces

…gs. Sync up slurm with new syntax

bgunnar5 · 2025-08-18T15:12:19Z

docs/Maestro/scheduling.md

+|  Multi-letter   | `--`           |  `=`          | `setopt: {foo: bar} | `--setopt=foo=bar`     |
+|  Boolean flag w/key   | as above       | as above      | `setopt: {foobar: } | `--setopt=foobar`      |


Missing backticks in the 4th columns here which is making the formatting on mkdocs all wonky

… through allocation and and launcher args

…flux directives

…her args

bgunnar5

Not sure if this branch is totally done yet but I had some time to do a review now.

bgunnar5 · 2025-09-22T13:50:59Z

maestrowf/abstracts/interfaces/schedulerscriptadapter.py

+        # Handle old scalar syntax which applied to allocatios only
+        if not isinstance(step_exclusive, dict):
+            return {
+                "allocation": step_exclusive,
+                "launcher": False,
+            }


Do we want to deprecate this in the future? If so, maybe we should add a warning log statement here to notify users.

That is a good question. Should we deprecate this shorthand? It is more often useful at the allocation level (scheduling to core scheduled partitions being one of the few use cases I can think of), so maybe useful to keep it. But I've no strong opinions on it either way.

bgunnar5 · 2025-09-22T13:55:45Z

maestrowf/interfaces/script/_flux/flux0_49_0.py

+        return cls.known_alloc_arg_types
+
+    @classmethod
+    def addtl_alloc_arg_type_map(cls, option):


May want to rename this to make it more obvious that this is a getter method. Maybe get_addtl_alloc_arg_type?

yeah, that's probably good for this version. In the subsequent PR's that will generalize this, something a little more descriptive like 'get_normalized..' or something? though with some iteration to make it not 30 char's long..

bgunnar5 · 2025-09-22T13:58:17Z

maestrowf/interfaces/script/_flux/flux0_49_0.py

+            # for av_name, av_value in arg_value.items():
+            #     value_str = render_arg_value(av_name, av_value)
+            #     yield "{prefix}{key}{sep}{value}".format(
+            #         prefix=arg_info['prefix'],
+            #         key=arg_key,
+            #         sep=arg_info['sep'],
+            #         value=value_str
+            #     )


Still needed? Same question goes for render_arg_value() above

bgunnar5 · 2025-09-22T14:00:10Z

maestrowf/interfaces/script/_flux/flux0_49_0.py

+            addtl_batch_args = {}
+
+        # May want to also support setattr_shell_option at some point?
+        for batch_arg_type in ["attributes", "shell_options", "conf"]:


use cls.addtl_alloc_arg_types instead of hardcoded list unless you have a specific reason for this?

bgunnar5 · 2025-09-22T14:02:57Z

maestrowf/interfaces/script/_flux/flux0_49_0.py

+                if conf_dict:
+                    LOGGER.warn("'conf' options not currently supported with "
+                                " nested=False.  Ignoring.")


is this a flux limitation or a limitation we're enforcing ourselves?

Not so much a limitation as part of the design. There's no argument for passing it in like with the from_nest and from_batch command methods, likely because unlike those two, this one doesn't create a nested broker to attach it to.

Longer term I think this option in Maestro will may get deprecated unless we find a good use case for it; quite different behavior from the from_nest_command option.

got it, that makes sense

bgunnar5 · 2025-09-22T14:32:27Z

maestrowf/interfaces/script/fluxscriptadapter.py

+        self._attr_prefixes = ['S', 'setattr']
+        self._opts_prefixes = ['o', 'setopt']
+        self._conf_prefixes = ['conf']  # No abbreviated form for this in flux docs


are these used? Seems like the same thing as the _allocation_args_map

bgunnar5 · 2025-09-22T14:47:28Z

maestrowf/interfaces/script/fluxscriptadapter.py

+            "nodes": f"{self._flux_directive}" + "-N {nodes}",
+            # NOTE: always use seconds to guard against upstream default behavior changes
+            "walltime": f"{self._flux_directive}" + "-t {walltime}s",
+            "queue": f"{self._flux_directive}" + "-q {queue}",
+            "bank": f"{self._flux_directive}" + "--bank {bank}",


don't think you need '+' in these strings; can just use f"{self._flux_directive} -N {nodes}" and similar entries for walltime, queue, and bank

It's more to defer them -> walltime, nodes, .. (and later queue/bank) are step specific values vs the flux_directive being adapter specific. The + joined strings get rendered later when writing out the batch script for individual steps.

bgunnar5 · 2025-09-22T14:50:57Z

maestrowf/interfaces/script/fluxscriptadapter.py

+                # TODO: add better mechanism for tracking whicn args
+                #       actually get used; dicts can't do this..


dataclass use-case? 👀

bgunnar5 · 2025-09-22T15:04:19Z

maestrowf/interfaces/script/fluxscriptadapter.py

+        # Handle the exclusive flags, updating batch block settings (default)
+        # with what's set in the step
+        step_exclusive_given = "exclusive" in step.run
+        step_exclusive = self._exclusive
+        if step_exclusive_given:
+            # Override the default with this step's setting
+            step_exclusive.update(self.get_exclusive(step.run.get("exclusive", False)))


nearly identical code is in get_header and get_parallelize_command. Consider moving logic to a method to avoid duplication.

bgunnar5 · 2025-09-22T15:10:04Z

tests/conftest.py

+
+        # Set up the output directory.
+        out_dir = environment.remove("OUTPUT_PATH")
+        if output_path:


should output_path be a default argument output_path=None since we have this check?

…eeding different formats

…n on failures

…gs ended up in completely different subtrees

… normalization of alloc args

…malization

bgunnar5

Looks good. A lot of these suggestions are just removing commented code and print statements. Really nothing major

bgunnar5 · 2025-10-21T14:15:37Z

docs/Maestro/scheduling.md

+|  **Key Type**         | **Prefix** | **Separator** | **Example YAML**     | **Example CLI Input/Directive** |
+|    :-                 | :-            | :-       |    :-                |       :-               |
+|  Single letter        |  `-`           | `" "` (space) | `o: {bar: 42}`       | `-o bar=42`            |
+|  Multi-letter         | `--`           |  `=`          | `setopt: {foo: bar}` | `--setopt=foo=bar`     |
+|  Boolean flag w/key   | as above       | as above      | `setopt: {foobar: }` | `--setopt=foobar`      |
+|  Boolean flag w/o key | as above       | as above      | `exclusive: `        | `--exclusive`      |


In the Example YAML Column here, you may want each of these to be multi-line instead like users are used to seeing? I've no strong opinion on this, just thinking from users POV.

I've done this in a table before with html tags. So for that first example it'd become:

<pre><code>o: bar: 42</code></pre>

Ugly in raw text but formats nicely in the table.

bgunnar5 · 2025-10-21T14:20:24Z

docs/Maestro/scheduling.md

+### Extra Flux Args
+----
+
+As of :material-tag:`1.1.12`, the flux adapter takes advantage of new argument pass through for scheduler options that Maestro cannot abstract away.  This is done via `allocation_args` and `launcher_args` in the batch block, which expand upon the previous `args` input which only applied to `$(LAUNCHER)`.  There are some caveat's here due to the way Maestro talks to flux.  The current flux adapters all use the python api's from Flux to build the batch jobs, with the serialized batch script being serialized separately instead of submitted directly as with the other schedulers.  A consequence of this is the `allocation_args` map to specific call points on that python api, and thus the option pass through is not quite arbitrary.  There are 4 currently supported options for allocations which cover a majority of usecases (open an issue and let us know if one you need isn't covered!):


Suggested change

As of :material-tag:`1.1.12`, the flux adapter takes advantage of new argument pass through for scheduler options that Maestro cannot abstract away. This is done via `allocation_args` and `launcher_args` in the batch block, which expand upon the previous `args` input which only applied to `$(LAUNCHER)`. There are some caveat's here due to the way Maestro talks to flux. The current flux adapters all use the python api's from Flux to build the batch jobs, with the serialized batch script being serialized separately instead of submitted directly as with the other schedulers. A consequence of this is the `allocation_args` map to specific call points on that python api, and thus the option pass through is not quite arbitrary. There are 4 currently supported options for allocations which cover a majority of usecases (open an issue and let us know if one you need isn't covered!):

As of :material-tag:`1.1.12`, the flux adapter takes advantage of new argument pass through for scheduler options that Maestro cannot abstract away. This is done via `allocation_args` and `launcher_args` in the batch block, which expand upon the previous `args` input which only applied to `$(LAUNCHER)`. There are some caveat's here due to the way Maestro talks to flux. The current flux adapters all use the python api's from Flux to build the batch jobs, with the serialized batch script being serialized separately instead of submitted directly as with the other schedulers. A consequence of this is the `allocation_args` map to specific call points on that python api, and thus the option pass through is not quite arbitrary. There are 4 currently supported options for allocations which cover a majority of usecases (open an issue and let us know if there's one you need that isn't covered!):

bgunnar5 · 2025-10-21T14:23:23Z

docs/Maestro/scheduling.md

+      resource.rediscover: "true"  # Use string "true" for Flux compatibility, not "True" or bool True
+  launcher_args:
+    setopt:
+      optiona:   # Boolean flag, no value needed


is this supposed to be "option a" or "optional"?

optiona; just a made up name for some random option to demonstrate the syntax mapping from yaml to batch script

bgunnar5 · 2025-10-21T14:24:59Z

docs/Maestro/scheduling.md

+#flux: --setattr=foobar=whoops
+#flux: --conf=resource.rediscover=true
+
+flux run -n 1 -N 1 -c 1 --setopt=optiona  myapplication


same question, "option a" or "optional"? From this context I'm assuming the former in which case we may want to change the naming convention for clarity?

bgunnar5 · 2025-10-21T14:39:31Z

maestrowf/abstracts/interfaces/flux.py

+        .. note::
+
+           Should we have an enum for these or something vs random strings?


I like this idea. Could probably help with safeguarding against unsupported options

bgunnar5 · 2025-10-21T15:39:23Z

maestrowf/interfaces/script/_flux/flux0_49_0.py

+        # if addtl:
+        #     args.append("-o")
+        #     args.append(",".join(addtl))


Suggested change

# if addtl:

# args.append("-o")

# args.append(",".join(addtl))

bgunnar5 · 2025-10-21T15:41:09Z

maestrowf/interfaces/script/fluxscriptadapter.py

+        if self._allocation_args:
+            self._allocation_args = self._interface.normalize_additional_args(self._allocation_args)
+            # pprint(f"{self._allocation_args=}")
+
+        if self._launcher_args:
+            self._launcher_args = self._interface.normalize_additional_args(self._launcher_args)
+            # pprint(f"{self._launcher_args=}")


Suggested change

if self._allocation_args:

self._allocation_args = self._interface.normalize_additional_args(self._allocation_args)

# pprint(f"{self._allocation_args=}")

if self._launcher_args:

self._launcher_args = self._interface.normalize_additional_args(self._launcher_args)

# pprint(f"{self._launcher_args=}")

if self._allocation_args:

self._allocation_args = self._interface.normalize_additional_args(self._allocation_args)

if self._launcher_args:

self._launcher_args = self._interface.normalize_additional_args(self._launcher_args)

bgunnar5 · 2025-10-21T15:45:18Z

maestrowf/interfaces/script/fluxscriptadapter.py

        #         "Flux URI must be specified in batch or stored in the "
        #         "environment under 'FLUX_URI'")

        # NOTE: Host doesn"t seem to matter for FLUX. sbatch assumes that the


This __init__ method is getting pretty large. May want to consider splitting into separate methods in the future

bgunnar5 · 2025-10-21T15:49:36Z

maestrowf/interfaces/script/fluxscriptadapter.py

+
+        alloc_eflags = [self._allocation_args.pop(ekey, None) for ekey in exclusive_keys]
+        if alloc_eflags:
+            if step_exclusive:


should this be if step_exclusive and self._exclusive['allocation'] like the launcher section below? If so, you may just want a common method for this shared logic

thanks for commenting on this one.. looking through the rest of the exclusive handling it appears these new modes aren't actually tested either which is troublesome, so let me go add that and verify the right thing's happening here.. the behaviors are a little different given old method exclusive = allocation only.. Might defer some of the refactoring to one of the followon's. I think more of this needs to be in the script adapter base class and just do more normalization upstream, and get it setup for a more layered approach to add on the step keys in the header/writescript/submit funcs and only handle the batch block source in the init.. ugh, i've made a mess, lol..

bgunnar5 · 2025-10-21T15:58:56Z

maestrowf/interfaces/script/fluxscriptadapter.py

+        # Normalize the allocation args to api flux.job.JobspecV1 expects
+        packed_alloc_args = self._interface.pack_addtl_batch_args(self._allocation_args)
+        # print(f"{normalized_alloc_args=}")
+
+        # Setup placholder for the queue/bank attributes if no already added by user
+        # in the allocation_args
+        # NOTE: should we flatten these treedict style?  conf looks like no treedict, but
+        # the others look like they support it even via python api
+        # if "system" not in normalized_alloc_args["attributes"]:
+        #     normalized_alloc_args["attributes"]["system"] = {}
+
+        # Add queue and bank
+        queue = self._batch["queue"]
+        if queue == "":
+            queue = None
+        bank = self._batch["bank"]
+        if bank == "":
+            bank = None
+
+        # if self._batch["queue"]:
+        #     normalized_alloc_args["attributes"]["system"]["queue"] = self._batch["queue"]
+        # if self._batch["bank"]:
+        #     # TODO: revisit whether it makes sense to add bank if queue is empty ->
+        #     #       nested brokers usually have neither, and bank falls through silently..
+        #     normalized_alloc_args["attributes"]["system"]["bank"] = self._batch["bank"]
+
+        # pprint(f"Packed alloc args for {step.name}:")
+        # pprint(packed_alloc_args)


Suggested change

# Normalize the allocation args to api flux.job.JobspecV1 expects

packed_alloc_args = self._interface.pack_addtl_batch_args(self._allocation_args)

# print(f"{normalized_alloc_args=}")

# Setup placholder for the queue/bank attributes if no already added by user

# in the allocation_args

# NOTE: should we flatten these treedict style? conf looks like no treedict, but

# the others look like they support it even via python api

# if "system" not in normalized_alloc_args["attributes"]:

# normalized_alloc_args["attributes"]["system"] = {}

# Add queue and bank

queue = self._batch["queue"]

if queue == "":

queue = None

bank = self._batch["bank"]

if bank == "":

bank = None

# if self._batch["queue"]:

# normalized_alloc_args["attributes"]["system"]["queue"] = self._batch["queue"]

# if self._batch["bank"]:

# # TODO: revisit whether it makes sense to add bank if queue is empty ->

# # nested brokers usually have neither, and bank falls through silently..

# normalized_alloc_args["attributes"]["system"]["bank"] = self._batch["bank"]

# pprint(f"Packed alloc args for {step.name}:")

# pprint(packed_alloc_args)

# Normalize the allocation args to api flux.job.JobspecV1 expects

packed_alloc_args = self._interface.pack_addtl_batch_args(self._allocation_args)

# Add queue and bank

queue = self._batch["queue"]

if queue == "":

queue = None

bank = self._batch["bank"]

if bank == "":

bank = None

… defaults. Add additional script test case and slurm script tests.

bgunnar5

Great work!

jwhite242 added 5 commits August 13, 2025 15:21

Update schema to support new exclusive syntax for separate launcher/a…

6e74f8d

…llocation control

Initial prototype of optional launcher/allocation args pass through h…

4bee81e

…andling in flux interfaces

Properly apply step exclusive key as override to batch/adapter settin…

4ccc470

…gs. Sync up slurm with new syntax

update method type on exclusive sanitizer

ea2d3b4

Initial documentation for new optional argument handling in batch block

cd632b9

bgunnar5 reviewed Aug 18, 2025

View reviewed changes

jwhite242 added 6 commits August 26, 2025 10:41

Fix stray refactoring debris

1e824e1

Fixtures and tests for verifying flux script writing correctly passes…

0a79699

… through allocation and and launcher args

Guard flux script writer to only run when flux is present

dfb0aea

Add helper for flattening dictionaries into dot syntax strings

4e70971

Update flux script test and test spec to use new dot string util for …

d450bbd

…flux directives

Add test for verifying live jobspecs get correct allocation and launc…

af62de2

…her args

bgunnar5 reviewed Sep 22, 2025

View reviewed changes

jwhite242 added 11 commits October 7, 2025 10:50

Add funcs for working with dotpath dicts/tuples and tests

567e51c

Add tests of non scalar leaf values in dotpath machinery

cebd217

Add value coercion for dotpath utils, recursive update

20d3409

Refactor additional arg handling to deal with python api end points n…

c923e74

…eeding different formats

Replace fail with assertionerror to avoid stopping entire test sessio…

5236603

…n on failures

Add extra info to failed jobspec tests to show full spec in case thin…

def5b65

…gs ended up in completely different subtrees

Finish wiring up queue/bank and update test specs/expected outputs

9bcc059

Move INFO lines to allow in step directives, fix test data to reflect…

93c05f0

… normalization of alloc args

Add docstrings/tests for dict utils

fb11228

Remove deepcopy comments

4e50a92

Misc tweaks and fixes, and update batch script outputs to reflect nor…

d4079bd

…malization

bgunnar5 approved these changes Oct 21, 2025

View reviewed changes

jwhite242 added 2 commits October 22, 2025 17:09

Rework exclusive handling for proper layering of steps ontop of batch…

25d7482

… defaults. Add additional script test case and slurm script tests.

Address review feedback to clarify examples

74c4361

bgunnar5 approved these changes Oct 23, 2025

View reviewed changes

		\| Multi-letter \| `--` \| `=` \| `setopt: {foo: bar} \| `--setopt=foo=bar` \|
		\| Boolean flag w/key \| as above \| as above \| `setopt: {foobar: } \| `--setopt=foobar` \|

		# TODO: add better mechanism for tracking whicn args
		# actually get used; dicts can't do this..

		.. note::

		Should we have an enum for these or something vs random strings?

	# if addtl:
	# args.append("-o")
	# args.append(",".join(addtl))

Extra Flux Options #459

Are you sure you want to change the base?

Extra Flux Options #459

Uh oh!

Conversation

jwhite242 commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgunnar5 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgunnar5 Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgunnar5 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgunnar5 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

jwhite242 commented Aug 15, 2025 •

edited

Loading

bgunnar5 Sep 22, 2025 •

edited

Loading