feat: ob create benchmark & module (Closes: #171) #172

btraven00 · 2025-08-22T10:33:06Z

Ticket

#171

Description

Add basic capabilities to create new benchmark and modules, using pre-defined templates.

Adding copier as the main engine for templating - uses jinja templates.

Checklist:

My code follows the code style of this project.
My CLI method respects the signature defined in the task.
I have documented the CLI method accordingly.
I have added tests to cover my changes.
I have added a CHANGELOG entry.

Remarks for the reviewer:

More interested in the dependencies and process than on feature completeness. We can refine later for 0.4.0

omnibenchmark/templates/module/README.md.jinja

btraven00 · 2025-08-22T12:00:59Z

I would not bother fixing these tests if they're gonna be revamped in #170

btraven00 · 2025-08-22T12:01:23Z

actually the yaml reading will need adaptation after merging #170 (do we want to land that today too?)

imallona · 2025-08-22T12:27:30Z

agree about being lenient with the tests. It passes the 'clustbench' integration test so 🆗 for me. As for merging #170 today it feels rushed, I wouldn't. @DanInci what do you think?

Tried ob create benchmark. Works nicely! Two unexpected behaviours:

I find a bit annoying you have to run it on an empty folder and then it tells you you need to cd there. You are there.
The resulting benchmark.yaml does not pass ob info topology -b benchmark.yaml because it says

(omnibenchmark) [imallona@here foo]$ ob info topology -b benchmark.yaml 
Error: Failed to parse YAML as a valid OmniBenchmark: File "benchmark.yaml", line 5, col 1:  Unknown argument: license = 'MIT'.

I guess it will fail with ob run benchmark for that reason as well?

Tried ob create module and works nicely as well! 💪 Couple of comments:

If ob create benchmark license was set to MIT by default, now it is set as GPLv3. Is this intended? And, in the future when running ob create module --bechmark yaml in the future, will it enforce the same license?
I tried to overwrite the module starting once with an R entrypoint and next a bash, and it warned me about overwriting the README, omnibenchmark.yaml, citation etc; but it didn't delete the (deprecated) R entrypoint. I guess this is not unexpected, but feels odd.
I find the input and output arguments used for parsing useful for copypasting/extending, but I guess users (Mark) should be prepared and know these should match the benchmarking yaml's and not be input nor output. If ensuring some arguments are provided for parsing purposes, I think current ob requires a module_name and output_dir (perhaps named differently) that are mandatory arguments no matter the module's nature. Perhaps worth templating those?

imallona

Please confirm ob run benchmark runs using the generated yaml even if containing a license stanza. ob info topology complains about yaml (in)validity and I wonder whether ob run would.

Otherwise looks great! Thanks a lot

imallona · 2025-08-22T12:51:39Z

CHANGELOG.md


+## [0.4.0](main) UNRELEASED
+
+- feat: add create benchmark & create module command (#171)


is this meant to be 0.4.0 or 0.3.1?

0.4.0 since brings new features

(if we want to release 0.3.1 for whatever reason I think it should be cut from dev into main as cherry-picks, or via a detached state from the release/0.3.0 branch)

btraven00 · 2025-08-22T16:18:41Z

fixed the license stuff, and one bug on the created identifiers.

ob info now correctly parses the topology:

❯ ob create benchmark
Starting copier questionnaire for new benchmark using OmniBenchmark v0.3.0.post10+dirty...
🎤 What is the name of your benchmark?
   my-awesome-benchmark
🎤 What is your name?
   Your Name
🎤 What is your email?
   [email protected]
🎤 Choose a license for your benchmark
   MIT
🎤 Brief description of your benchmark
   A brief description of what this benchmark does
🎤 Choose the default software environment backend
   conda

Copying from template version None
    create  CITATION.cff
    create  validations.yaml
    create  envs
    create  envs/conda.yaml
    create  .gitignore
    create  README.md
    create  benchmark.yaml


Initialized Git repository
Created initial commit
Benchmark scaffolding created successfully with OmniBenchmark v0.3.0.post10+dirty
Next steps:
  1. Review and customize the benchmark configuration
  2. Add your benchmark modules
  3. Run 'ob validate' to check your configuration (coming soon!)
❯ ob info topology -b benchmark.yaml
---
title: my-awesome-benchmark
---
flowchart LR
	classDef param fill:#f96
	subgraph datasets
		dataset_loader
	end
	subgraph methods
		cool_method_1
		dataset_loader --> cool_method_1
	end
	subgraph metrics
		strict_metric_1
		cool_method_1 --> strict_metric_1
	end
	subgraph params_dataset_loader
		426fc04f04bf8fdb5831dc37bbb6dcf70f63a37e05a68c6ea5f63e85ae579376['--foo', 'bar']
	end
	params_dataset_loader:::param --o dataset_loader
	subgraph params_cool_method_1
		426fc04f04bf8fdb5831dc37bbb6dcf70f63a37e05a68c6ea5f63e85ae579376['--foo', 'bar']
	end
	params_cool_method_1:::param --o cool_method_1
	subgraph params_strict_metric_1
		426fc04f04bf8fdb5831dc37bbb6dcf70f63a37e05a68c6ea5f63e85ae579376['--foo', 'bar']
	end
	params_strict_metric_1:::param --o strict_metric_1

I tried to overwrite the module starting once with an R entrypoint and next a bash, and it warned me about overwriting the README, omnibenchmark.yaml, citation etc; but it didn't delete the (deprecated) R entrypoint. I guess this is not unexpected, but feels odd.

Yea, I feel very unsafe to try to delete files. This should be a one-off operation.

I find a bit annoying you have to run it on an empty folder and then it tells you you need to cd there. You are there.

Fixed, doesn't hint to cd if target == cwd.

FWIW you can also instantiate passing a path: create benchmark /foo/bar

I find the input and output arguments used for parsing useful for copypasting/extending, but I guess users (Mark) should be prepared and know these should match the benchmarking yaml's and not be input nor output. If ensuring some arguments are provided for parsing purposes, I think current ob requires a module_name and output_dir (perhaps named differently) that are mandatory arguments no matter the module's nature. Perhaps worth templating those?

I think we'd be better off by trying to design a good contract for what modules are expected to receive. I am not even sure out_dir is a generalizable abstraction - I feel we're designing the grammar too closely to snakemake at this point.

The whole argument parsing is very ad-hoc at this point, it might be that we do not need it at all if we pass arguments via the snakemke object.

Shall we try to discuss the mandatory arguments and the rules to generate the mapping in a different issue at this point?

Add basica capabilities to create new benchmark and modules, using pre-defined templates. Adding copier as the main engine for templating - uses jinja templates. In the future we can dettach the module logic from omnibenchmark embedded (or online) repositories, and allow for specific logic defined by each benchmark. For now a lot of this is stub. However, it poses an interesting design problem: how much information needs to be transferred from the stage level to the module. One important thing to do will be to template example tests - as I try to do in this commit for demonstration. We could take these tests from the validation specification - to be landed in the 0.4.x release cycle. This would give us nice post-execution validation paths, and possibly also the possibility to "test-drive" a module with, e.g., datasets+preprocessing that are taged as "example", or even to allow automation in the sketched "ob try module ..." - since we can codegen the population of the QA tests from the information contained in the benchmark.yaml. That said, a lesson learned from doing this PR is that we might be lacking common structural information about the "contract" that all modules in a given stage need to satisfy. Right now, the passing of parameters is done on a module level - there's no interface to enforce that is common to all modules in the same stage. If we had, we could add part of this discovery mechanism to the templating logic itself. To be discussed. fixes conditional template

btraven00 · 2025-08-22T16:27:39Z

If ob create benchmark license was set to MIT by default, now it is set as GPLv3. Is this intended? And, in the future when running ob create module --bechmark yaml in the future, will it enforce the same license?

For clarity, I "fixed" that it's choosing the same default now (MIT) for benchmark and module.

Whether the module has to follow the same license than the benchmark: I think the easiest thing to do is not to ask for input, if given a reference benchmark definition. I am not sure it's a legal problem, but itsure simplifies things to have all modules abiding by the same license (it can be tricky because we're in the same situation as debian packaging etc: is it the license of the contribution, or the license of the software? in that case and if we care, we need to be prepared to deal with licensing exceptions, compatibility etc - and possible capture realities with >1 license).

Again, possibly too much complexity to begin with, sanest would be to let the benchmark decide on the global license and the template follow. All can be manually overriden.

At the metadata level, I would only care about consistency between the different possible fields (CFF, LICENSE file if written too etc). Seems more of a task for the validator.

imallona

Thanks for fixing the license so quickly! Lgtm

DanInci

Really great work ! Both commands are intuitive to use.

I experienced just one issue with the software backends. Whenever choosing a conda environment for the benchmark / module, everything works fine, because a template conda.yml is created.

However, that's not the case with apptainer, or envmodule. The generated benchmark doesn't pass validation because its missing the backend file. I suggest at least touching the required file, if templating is not possible.

Error: An unexpected error occurred: Software environment path for 'apptainer' does not exist: 'test/envs/example.sif'.

DanInci · 2025-08-29T10:57:18Z

docs/src/howto.md


+Start a new benchmark:
+
+```bash
+ob create benchmark ~/my_benchmark
+```
+
+Creat a new module:
+
+```bash
+ob create module ~/my_new_module
+```
+
+
 For detailed instructions, see below.



I believe For detailed intructions, see below. got lost between the newly added commands. It should be before them.

Also, is it worth mentioning more about the ob create commands?

no, my intention was to leave those two hints as part of the quick start (maybe assuming that starting a new bench/module is what people will want to use first when starting on the project).

ob create, I feel cli reference might be enough? for sure it can be integrated into a beginner tutorial.

DanInci · 2025-08-29T11:03:44Z

...ark/templates/benchmark/envs/{% if software_backend == 'conda' %}conda.yaml{% endif %}.jinja

+name: {{ benchmark_name | lower | replace(' ', '_') | replace('-', '_') }}_env
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python>=3.9
+  - pip
+  - pip:
+    - # Add your pip dependencies here


Surprised this is an acceptable if statement in the file name. Looks so weird.

it looks very weird instead - but apparently it's valid since jinja pre-process templates in filenames. I will try to find cleaner-looking ways, but possibly that'll involve having switch statements and renames in a control file...

DanInci · 2025-08-29T11:07:00Z

omnibenchmark/templates/module/{{ entrypoint }}.jinja

+{%- if entrypoint == "run.py" -%}
+#!/usr/bin/env python3
+
+import argparse
+import sys
+from pathlib import Path
+
+# Add src directory to Python path
+src_dir = Path(__file__).parent / "src"
+sys.path.insert(0, str(src_dir))
+
+from main import process_data
+
+def parse_args():


Why do we keep both the entrypoint.py and the src/run.py as a template?
I feel like there should be just one entrypoint to not add more confusion. The user / module contributor should be able to structure their own project, they will need just the wrapper for their method.

I don't understand this comment. there is only one entrypoint template; depending on the name chosen in the wizard this is materialized onto an R or python script. the idea is to provide them with the boring boilerplate, like argument parsing etc, and maybe expand the templates into something more useful in the future.

DanInci · 2025-08-29T11:11:02Z

omnibenchmark/cli/create.py

+        # Initialize git repository if it doesn't exist
+        if not git_dir.exists():
+            subprocess.run(
+                ["git", "init"], cwd=target_path, check=True, capture_output=True
+            )
+            logger.info("Initialized Git repository")
+
+        # Add all files and create initial commit
+        subprocess.run(
+            ["git", "add", "."], cwd=target_path, check=True, capture_output=True
+        )


We still have the dependency on "GitPython>=3.1.0". We should be consistent, and either use it, or remove it entirely from the codebase.

btraven00 · 2025-09-19T10:14:41Z

However, that's not the case with apptainer, or envmodule. The generated benchmark doesn't pass validation because its missing the backend file. I suggest at least touching the required file, if templating is not possible.

Error: An unexpected error occurred: Software environment path for 'apptainer' does not exist: 'test/envs/example.sif'.

I see what's happening. We can template a dummy .eb or apptainer declaration; however, touching the binary modules or container image is going to be even more confusing as an error message. At some point compilation needs to happen, and that's up to the user.

btraven00 force-pushed the feat/template branch from 440bdb6 to 22add4e Compare August 22, 2025 10:36

imallona reviewed Aug 22, 2025

View reviewed changes

omnibenchmark/templates/module/README.md.jinja Show resolved Hide resolved

btraven00 requested review from imallona and DanInci August 22, 2025 10:42

imallona added a commit to omnibenchmark/clustering_example that referenced this pull request Aug 22, 2025

Try omnibenchmark/omnibenchmark#172

7051cb7

imallona requested changes Aug 22, 2025

View reviewed changes

imallona reviewed Aug 22, 2025

View reviewed changes

btraven00 force-pushed the feat/template branch from 22add4e to 45eef51 Compare August 22, 2025 16:21

imallona approved these changes Aug 22, 2025

View reviewed changes

DanInci reviewed Aug 29, 2025

View reviewed changes

refactor: use GitPython instead of the porcelain

7028e7c

feat: add skeleton for environment files

4da3602


		## [0.4.0](main) UNRELEASED

		- feat: add create benchmark & create module command (#171)

feat: ob create benchmark & module (Closes: #171) #172

Are you sure you want to change the base?

feat: ob create benchmark & module (Closes: #171) #172

Uh oh!

Conversation

btraven00 commented Aug 22, 2025

Ticket

Description

Checklist:

Remarks for the reviewer:

Uh oh!

Uh oh!

btraven00 commented Aug 22, 2025

Uh oh!

btraven00 commented Aug 22, 2025

Uh oh!

imallona commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imallona left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btraven00 commented Aug 22, 2025

Uh oh!

btraven00 commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imallona left a comment

Choose a reason for hiding this comment

Uh oh!

DanInci left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btraven00 Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btraven00 commented Sep 19, 2025

Uh oh!

Uh oh!

imallona commented Aug 22, 2025 •

edited

Loading

imallona left a comment •

edited

Loading

btraven00 commented Aug 22, 2025 •

edited

Loading

btraven00 Sep 19, 2025 •

edited

Loading