_find_duplicates and _is_url are helper methods that I've seen declared in a couple of places already. Let's factor them out.

you're right, they're duplicated in model.validation and model.benchmark. TBD

Why not move then the whole validation logic in the Validator superclass?

I suspect we want to leave just syntactic validation at this level (pydantic model for syntax), and have all the consistency checks (at the benchmark level) plus the module metadata and result validation under an unifying validators module.

I would move the model package under benchmark, since it anyways contains models related to the benchmark.

as we discussed, my plan is to move some interfaces, exceptions etc from benchmark to core, to avoid circular dependencies between validators, model and "benchmark" (in essence, "benchmark" should be very minimal after further refactoring between modules)

Really nicely structured and tested !
This will cover all use cases that we have so far. Thanks

wouldn't this run twice when PR-ing from dev to main?

I wonder whether there is any push, other than to master, that needs to be checked without a PR / review first

  pull_request:
  push:
    branches:
      - main

also, would a workflow_dispatch: make sense?

wouldn't this run twice when PR-ing from dev to main?
run twice: yes, I think this is pretty standard no? You want to test before (PR discussion) and after you merge (because some times, like in this same PR, squashing or merging introduces errors, mostly due to bad resolution of conflicts or bad rebasing etc.

Re. merging from dev to main, same: we're interested in catching divergence when tagging on main and doing hotfixes that might not be reconciled properly with dev.

tldr; I think it does no harm

isn't that much more going on, e.g. the test framework simplification?

difference between benchmarker and author?

path to what?

difference between these two? Same kind of comments apply to the rest of the example dict

Most things (arguments, parameters) react well to a 1 value, would it make more sense to set the defaults as 0s, 0.0s, Falses instead, so we reduce chances they are meaningful?

unrelated to this update but related conceptually. Does this ob digest alternatives such as:

• full commit lengths (Mark attempted to use full commits)
• branch names
• tags?

hmm afaik we enforce commits and not refs like branch/tag, I understood that was intentional (by the omni-schema spec). Re. long-name, I think we should be permissive yes.

thanks. I think it would be good to have an (undocumented) branch name spec for fast prototyping. But defeats the whole repro purpose, so we should ask the team again

hmm maybe it's something that we can allow w/ validation rules --strict

Is this meant so apptainer uses a docker image directly?

that, and accounting for our new plugin that does allow docker execution via podman/udocker

hmm, then we perhaps should get rid of ORAS (protocol) constraints if they're still in place. If using standard snmk plugins we can get images from anywhere, e.g. dockerhub and so on

full path or basename? meant to make sense of the robotspath and the easybuild config

are they paths or basenames?

this is a superimportant point, because it has to do with the .snakemake folder and so, beyond the output dir. Shall we document that? I'm thinking of some users wanting to have their YAML git tracked somewhere in their home, but the ob run on a scratch or similar. They might want to copy (symlink?) the YAML to the scratch and run ob there, right?

hmm, self.nodes is a Set, so this iteration is not deterministic. Which the point of this function, I know it feels I'm making no sense, and perhaps it's nonsense, but. You're trying to sort the nodes. But in order to sort the nodes, wouldn't it be safer to do so after pre-sorting (not topologically) them with sorted(self.nodes) or the similar? To avoid future poltergeists with complex DAGs with plenty of excludes, intermediate metric collectors etc, where randomly accessing the (node) set might be adding unwanted extra freedom.

the only purpose of this module is to drop networkx support, that was adding significantly to startup time.

You're trying to sort the nodes

this PR does not introduce any new behavior, it just mimicks the parts of the networkx API that we were already using.

that said, I think this discussion is linked to how we want to deal with the intermediate representation of the config file and the workflow-specific serialization of the blocks (snakemake or others).

in a sense, all the DAG parsing is redundant since snakemake will build its own. However, I would argue that we need simple DAG operations to have a canonical representation and validation of the tree(s) being loop free. Otherwise we're letting snakemake (or backend X) blow up errors directly to the user.

I don't think the snmk trick we're using (nesting results of B as a folder of its parent A) can give rise to loops. Perhaps this has to do with the mermaid/layout plotting only?

what does "not defined" mean, "unavailable" perhaps? or "unloadable"? Worth reporting the modulepath then? So the user has more info/debugging opportunities

I think unavailable or not loadable speaks about the ability to load them (maybe because the path cannot be resolved).

in this case, they're missing from the definition (or declaration, in the C sense). would that work better?

Hmm, something like this?

Error Missing software environment. Module X is not defined within the benchmarking YAML: it should be listed as part of the stanza software_environments within the benchmarking YAML header.

Generally, stanzas for envmodules are shaped as follows (e.g., for a module loadable with module load bwa/0.7.17)

software_environments:
  bwa:
    description: "a bwa module"
    envmodule: "bwa/0.7.17"

Or something like that?

Unless the error had to do with being unloadable. If so, the error might better report the trace of the failed module purge; module load command instead, as well as the modulepath?

I think this error is not very clear, is this capturing the "conda.yml" is not available at the given path? if so the user would benefit from "Failure to find a conda yaml at ./path/to/file as required to run module/metric collector X using conda"

no, it's just doing consistency check.

since we're here: I think there's a possible easy simplification of this whole logic instead of so many if.. else.
instead of managing conda, apptainer etc as different fields in the environment, I think what we need is type-checking a single entity, environment.

available_environments = List[Environment]

(possibly qualified by checking paths for existence, but that can come up later in the validation process)

where any((environment.type == benchmark.backend for environment in declared_environments)) needs to be True for agiven benchmark definition.

but how can we typecheck a module name, which only has a meaning after sourcing a bash function (e.g., the module load X verb for X), or that it's just a file (conda yaml, because snmk does generate the env itself)? Either we evaluate whether the YAML has the specification (so it's not empty), or whether it works (e.g., the module is loadable, the conda yml is well formed, the apptainer is present or downloadable etc).

Or we could go for validating files, e.g. looking for the module lua file, the conda env yml, or the apptainer local/remote image?

Again this message is not very clear, is the image (local? oras?) unavailable? or is it the yaml stanza missing? what does it mean to discuss docker and apptainer separately?

(meaning, I'd make the traces longer/more verbose, because these user errors are likely to happen during the first ob usage attempts so we'd better be extra informative there)

@btraven00 I'm reading the versioning, do you have some design / thoughts document somewhere? It might be it has been shared and I can't find it... thanks!