* Rename config

* Rename config

* Add Optuna config

* Simplify init of optuna study

* Update

* Small fix

* feat(skopt): more variants

* Add more kwargs for DEHB

* Add paper source

* Add task info for dummy problem

* Update HEBO config

* Rework Skopt config

* Use base config for synetune

* Add comment

* restrict ConfigSpace for now (yahpo and smac struggle)

* Clean SMAC configs

* Remove smac1.4 configs

(clash with ConfigSpace, would only work with containers)

* Update optuna configs

* Update Nevergrad configs / clean

* Update CHANGELOG.md

* Update

* Generate pymoo problems

* Update pymoo problems

* Update CHANGELOG.md

* feat(benchmark): add bbob 20d 100 vizier

* fix(hebo): sobol scrambling seed default

* refactor(hebo): use global seed for sobol

* refactor(hebo): update config to pass global seed to scramble

* feat(notebook): add comparison of SMAC, HEBO, RS on BBOB20d

* Update CHANGELOG.md

* feat(Ax): Add Ax optimizer (SO + MO)

* Update CHANGELOG.md

* feat(Ax): Type refinements, fixed OrdinalHyperparameter, added note regarding random seed

---------

Co-authored-by: thibautklenke <[email protected]>
Co-authored-by: benjamc <[email protected]>

* Ignore more

* Update

* Add notebook

* Update

* feat: validate ranking

* Update

* add notebook

* Update

* Do not plot

* Update

* Update notebook

* Format

* Compare subselections

* Also save run cfgs

* Update notebook

* fix timestamp

* goodbye bash scripts

* make prettier

* goosbye old file

* goodbye

* build(pyproject): update ruff setting

* Ignore more

* update program

* feat(subselect): finally as python

* tiny fixes

* refactor(subselection)

* fix(problem): generate pymoo config

* local parallel

* refactor(create_subset_config): del dir, better error msg, new cmd

* feat: new subsets for BB + MO

* run more pymoo problems

* Rename problem

* refactor(generate_problems): MO

* refactor(benchmark_footprint): notebook

* refactor(inspect_problems): update notebook

* build(yahpo_install): update

* update yahpo install

* update notebooks

* revert subselection progress

moved to branch feat/subselection

* please pandas

* please pandas

* fix(hebo): ordinal hyperparameter + precommit

* refactor(yahpo): update ConfigSpace API

* ignore more

* add scenario and subset_id to config

* add scenario and subset id to subselection configs

* remove duplicates in config

* refactor(gather_data): more config keys

* feat(gather_data): calculate log performance

* update notebook

* feat(report)

* fix(autorank): api

* ignore more

* feat(color_palette): more and nicer colors

* feat(generate_report): silent plotting

* fix(ax): conversion of ordinal HPs

* style(utils): pre-commit

* style(...): pre-commit

* style

* fix(ax): cat HP

* refactor(generate_report)

* fix(ax): allow_inactive_with_values=True

* fix(yahpo): ConfigSpace deprecation warning

* docs(generate_report): more info

* fix(run_autorank): if nothing is lost

* fix(utils): filter by final performance: if there is a small max val correctly limit

* feat(gather_data): collect from several folders

* refactor(generate_report): goodbye plots, fix ranks and norm

* style(file_logger)

* style(pareto_front)

* style(overriderfinde)

* style(loggingutils)

* style(index_configs)

* refactor(generate_report): fun arg default

* fix(run_autorank): log msg pos

* pre-commit 1

* refactor/fix(nevergrad): pass trial seed to trial info

* Remove DummyOptimizer because Random Search behaves the same

* Remove DummyOptimizer because Random Search behaves the same

* style: make ruff happy

* refactor(task): set n_objectives as 1 per default

* refactor(synetune): set max budget (fidelity) as fixed hp in synetune

* Please mypy nr 1

* refactor(install_yahpo): control root of carps

* fix(task): dataclass unmutable arg

* fix(task): dataclass unmutable arg

* bbob: update ioh requirements

* style: please mypy

* Update CHANGELOG.md

* refactor(objective_functions): rename folder, rename file

* refactor: Problem -> ObjectiveFunction

* refactor: import objective_function

* refactor: problem_id -> task_id

* refactor: carps.benchmarks -> carps.objective_functions

* refactor(task)

* refactor(installation)

* refactor(install): update

* current

* refactor(BREAKING): new task definition

* refactor: make task instantiable

* refactor: generate tasks, new configs

* refactor(optimizer): add metadata about mo/mf

* tests(optimizers): add

* testing

* fix install scripts

* refactor(dummy_problem): handle ListConfig

* update dependencies

* refactor(Makefile): more commands

* refactor: rename budget -> fidelity

* refactor: rename problem -> task or objective function

* fix(smacconfig)

* format(notebooks): ruff

* format

* refactor: scenario -> task_type

* refactor(hebo): deactivate due to install issues

* fix(hpob): path to hpob surrogate files

* ignore more

* fix(hpobench_config)

* fix(pymoo_config)

* refactor(yahpo_tasks): regenerate

* fix(hpobench): installation (container building)

* fix(hpobench): configs, build container

* refactor(hpobench): configs

* refactor(subselection): update config files

* refactor(hpobench_configs): tab -> tabular

* refactor(subselection_blackbox): update to new tasks

* build: update requirements

* Update carps version

* Update CHANGELOG.md

* add todo

* build/refactor(installation): use make commands from anywhere

* Update README.md

* refactor(install): benchmarks with data

…ve python objects

…ve python objects

…assing to objective function evlauate

* fix(gather_data): dict accesses, lists, task_id

* fix(create_cluster_configs): dict accesses

* feat(scrape_to_db): Scrape file-logged results to database

* Update README.md

* Sleeker type handling and happy pre-commit :)

* Refactor

* refactor: subselection notebooks

* refactor(run_autorank): directly pass df crit

* build(pyproject): remove all extras

* update notebook

* fix(legacy_task_ids)

* feat: create database experiments via slurm

* refactor/feat: update subselection, add python methods

* format(shift_v2nobrute.c): finally

* feat(subselect): make usable

* fix(yahpo): generation of configs, set time budget to none

* refactor(create_cluster_configs): calculate config hash differently

* feat(create_cluster_configs_subselection)

* fix(file_logger): logging msg

* fix(yahpo_task_gen): output space

* fix(yahpo output space)

* feat(show_stats): split errors into yahpo and non yahpo, save error message

* fix/build: hpobench benchmark install

* refactor(database_logger): also log n function calls for incumbents

* refactor(database_logger): also log n function calls for incumbents

* feat: download and process results from database

* feat(show_stats): split error msgs in known and unknown

* fix(nevergrad): optimizer config

(one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up)

* refactor(Makefile): for using uv, make sure pip is installed and accessible

* fix(synetune): passing metrics

* fix(nevergrad): config out of bounds (just a tiny bit)

* refactor(create_cluster_configs_subselection)

* fix(Ax): always convert cost to float

* fix(Ax): hopefully numeric mean bounds

* feat(reset_experiments): reset only yahpo attr none type errors

* ignore more

* refactor(show_stats): filter more errors

* refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts

* refactor(create_cluster_configs): rename variables

* refactor(gather_data): handle logs from database

* docs(process_logs): add runcommand

* feat(reset_experiments): add more reset options

* fix(reset_experiments): correctly delete falsely done experiments

* refactor(generate_report)

* refactor(generate_report): small plotting adjustments

* fix(run_autorank):  class args

* fix(yahpo): 1st to fix yahpo nonetype error

Copy onnx model.

* refactor(check_missing): adapt to new structure

* refactor(gather_data): adapt to new structure

* refactor(pyproject.toml): add pyexperimenter requirement

* refactor(scrape_results_to_database)

* refactor(Makefile): upgrade numpy

* feat(container): add recipe for virtual env

* refactor(yahpo install)

* refactor(requirements)

* refactor(README): adjust paths

* refactor(download_results): add more infos

* docs(README): add info about container

* refactor: subselection notebooks

* refactor(run_autorank): directly pass df crit

* build(pyproject): remove all extras

* update notebook

* fix(legacy_task_ids)

* feat: create database experiments via slurm

* refactor/feat: update subselection, add python methods

* format(shift_v2nobrute.c): finally

* feat(subselect): make usable

* fix(yahpo): generation of configs, set time budget to none

* refactor(create_cluster_configs): calculate config hash differently

* feat(create_cluster_configs_subselection)

* fix(file_logger): logging msg

* fix(yahpo_task_gen): output space

* fix(yahpo output space)

* feat(show_stats): split errors into yahpo and non yahpo, save error message

* fix/build: hpobench benchmark install

* refactor(database_logger): also log n function calls for incumbents

* refactor(database_logger): also log n function calls for incumbents

* feat: download and process results from database

* feat(show_stats): split error msgs in known and unknown

* fix(nevergrad): optimizer config

(one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up)

* refactor(Makefile): for using uv, make sure pip is installed and accessible

* fix(synetune): passing metrics

* fix(nevergrad): config out of bounds (just a tiny bit)

* refactor(create_cluster_configs_subselection)

* fix(Ax): always convert cost to float

* fix(Ax): hopefully numeric mean bounds

* feat(reset_experiments): reset only yahpo attr none type errors

* ignore more

* refactor(show_stats): filter more errors

* refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts

* refactor(create_cluster_configs): rename variables

* refactor(gather_data): handle logs from database

* docs(process_logs): add runcommand

* feat(reset_experiments): add more reset options

* fix(reset_experiments): correctly delete falsely done experiments

* refactor(generate_report)

* refactor(generate_report): small plotting adjustments

* fix(run_autorank):  class args

* fix(yahpo): 1st to fix yahpo nonetype error

Copy onnx model.

* refactor(check_missing): adapt to new structure

* refactor(gather_data): adapt to new structure

* refactor(pyproject.toml): add pyexperimenter requirement

* refactor(scrape_results_to_database)

* refactor(Makefile): upgrade numpy

* feat(container): add recipe for virtual env

* refactor(yahpo install)

* refactor(requirements)

* refactor(README): adjust paths

* refactor(download_results): add more infos

* docs(README): add info about container

* refactor: subselection notebooks

* refactor(run_autorank): directly pass df crit

* build(pyproject): remove all extras

* update notebook

* fix(legacy_task_ids)

* feat: create database experiments via slurm

* refactor/feat: update subselection, add python methods

* format(shift_v2nobrute.c): finally

* feat(subselect): make usable

* fix(yahpo): generation of configs, set time budget to none

* refactor(create_cluster_configs): calculate config hash differently

* feat(create_cluster_configs_subselection)

* fix(file_logger): logging msg

* fix(yahpo_task_gen): output space

* fix(yahpo output space)

* feat(show_stats): split errors into yahpo and non yahpo, save error message

* fix/build: hpobench benchmark install

* refactor(database_logger): also log n function calls for incumbents

* refactor(database_logger): also log n function calls for incumbents

* feat: download and process results from database

* feat(show_stats): split error msgs in known and unknown

* fix(nevergrad): optimizer config

(one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up)

* refactor(Makefile): for using uv, make sure pip is installed and accessible

* fix(synetune): passing metrics

* fix(nevergrad): config out of bounds (just a tiny bit)

* refactor(create_cluster_configs_subselection)

* fix(Ax): always convert cost to float

* fix(Ax): hopefully numeric mean bounds

* feat(reset_experiments): reset only yahpo attr none type errors

* ignore more

* refactor(show_stats): filter more errors

* refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts

* refactor(create_cluster_configs): rename variables

* refactor(gather_data): handle logs from database

* docs(process_logs): add runcommand

* feat(reset_experiments): add more reset options

* fix(reset_experiments): correctly delete falsely done experiments

* refactor(generate_report)

* refactor(generate_report): small plotting adjustments

* fix(run_autorank):  class args

* fix(yahpo): 1st to fix yahpo nonetype error

Copy onnx model.

* refactor(check_missing): adapt to new structure

* refactor(gather_data): adapt to new structure

* refactor(pyproject.toml): add pyexperimenter requirement

* refactor(scrape_results_to_database)

* refactor(Makefile): upgrade numpy

* feat(container): add recipe for virtual env

* refactor(yahpo install)

* refactor(requirements)

* refactor(README): adjust paths

* refactor(download_results): add more infos

* docs(README): add info about container

* fix(unique task ids)

* fix(dummy_problem): passing of configuration space

* fix(synetune): passing metric for SyncMOBSTER

* fix/build(mfpbench): new xgboost for pd1

* fix(test_tasks): query fidelity

* fix(pyproject): ONLY COPY CARPS FILES 😭👽

* fix(pyproject.toml) email adress

* fix/build: properly include files in package (not too many)

* update README.md

* docs(reset_experiments): add convenience cmds in docstring

* refactor(show_stats): print number of known errors

* fix(Makefile): access works

Makefile did not get copied into

* move container_recipes

* style(prepare_nas_benchmarks): add docstr

* refactor: rename container recipe paths

* fix: install paths

* fix(install_yahpo): pip install

* build(pyproject.toml): include recipe data

* Update README.md

* Update CHANGELOG.md