Tags: automl/CARP-S
Tags
v1.0.4 (#205) * Fix HPOBench (#158) * Fix tabular benchmark not working * Update hpobench install * Add script to fix nas container error * Update CHANGELOG.md * Optuna: add new MO config (#160) * Rename config * Rename config * Add Optuna config * Simplify init of optuna study * Update * Small fix * Update optimizer configs (#161) * feat(skopt): more variants * Add more kwargs for DEHB * Add paper source * Add task info for dummy problem * Update HEBO config * Rework Skopt config * Use base config for synetune * Add comment * restrict ConfigSpace for now (yahpo and smac struggle) * Clean SMAC configs * Remove smac1.4 configs (clash with ConfigSpace, would only work with containers) * Update optuna configs * Update Nevergrad configs / clean * Update CHANGELOG.md * More Pymoo Problems (#162) * Update * Generate pymoo problems * Update pymoo problems * Update CHANGELOG.md * Update HEBO & BBOB-Vizier (#163) * feat(benchmark): add bbob 20d 100 vizier * fix(hebo): sobol scrambling seed default * refactor(hebo): use global seed for sobol * refactor(hebo): update config to pass global seed to scramble * feat(notebook): add comparison of SMAC, HEBO, RS on BBOB20d * Update CHANGELOG.md * Merge * feat(Ax): Add Ax optimizer (SO + MO) (#166) * feat(Ax): Add Ax optimizer (SO + MO) * Update CHANGELOG.md * feat(Ax): Type refinements, fixed OrdinalHyperparameter, added note regarding random seed --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> * fix: SMAC callbacks (#167) * Update CHANGELOG.md * fix(ax): conversion of ordinal HPs * fix(ax): cat HP * fix(ax): allow_inactive_with_values=True * fix(smac20): remove unrecognized arg from TrialValue * fix(ax): typos * Research/subset (#164) * Ignore more * Update * Add notebook * Update * feat: validate ranking * Update * add notebook * Update * Do not plot * Update * Update notebook * Format * Compare subselections * Also save run cfgs * Update notebook * fix timestamp * goodbye bash scripts * make prettier * goosbye old file * goodbye * build(pyproject): update ruff setting * Ignore more * update program * feat(subselect): finally as python * tiny fixes * refactor(subselection) * fix(problem): generate pymoo config * local parallel * refactor(create_subset_config): del dir, better error msg, new cmd * feat: new subsets for BB + MO * run more pymoo problems * Rename problem * refactor(generate_problems): MO * refactor(benchmark_footprint): notebook * refactor(inspect_problems): update notebook * build(yahpo_install): update * update yahpo install * update notebooks * revert subselection progress moved to branch feat/subselection * please pandas * please pandas * fix(hebo): ordinal hyperparameter + precommit * refactor(yahpo): update ConfigSpace API * ignore more * add scenario and subset_id to config * add scenario and subset id to subselection configs * remove duplicates in config * refactor(gather_data): more config keys * feat(gather_data): calculate log performance * update notebook * feat(report) * fix(autorank): api * ignore more * feat(color_palette): more and nicer colors * feat(generate_report): silent plotting * fix(ax): conversion of ordinal HPs * style(utils): pre-commit * style(...): pre-commit * style * fix(ax): cat HP * refactor(generate_report) * fix(ax): allow_inactive_with_values=True * fix(yahpo): ConfigSpace deprecation warning * docs(generate_report): more info * fix(run_autorank): if nothing is lost * fix(utils): filter by final performance: if there is a small max val correctly limit * feat(gather_data): collect from several folders * refactor(generate_report): goodbye plots, fix ranks and norm * style(file_logger) * style(pareto_front) * style(overriderfinde) * style(loggingutils) * style(index_configs) * refactor(generate_report): fun arg default * fix(run_autorank): log msg pos * pre-commit 1 * refactor/fix(nevergrad): pass trial seed to trial info * Remove DummyOptimizer because Random Search behaves the same * Remove DummyOptimizer because Random Search behaves the same * style: make ruff happy * refactor(task): set n_objectives as 1 per default * refactor(synetune): set max budget (fidelity) as fixed hp in synetune * Please mypy nr 1 * refactor(install_yahpo): control root of carps * fix(task): dataclass unmutable arg * fix(task): dataclass unmutable arg * bbob: update ioh requirements * style: please mypy * Update CHANGELOG.md * 170 update nomenclature (#175) * refactor(objective_functions): rename folder, rename file * refactor: Problem -> ObjectiveFunction * refactor: import objective_function * refactor: problem_id -> task_id * refactor: carps.benchmarks -> carps.objective_functions * refactor(task) * refactor(installation) * refactor(install): update * current * refactor(BREAKING): new task definition * refactor: make task instantiable * refactor: generate tasks, new configs * refactor(optimizer): add metadata about mo/mf * tests(optimizers): add * testing * fix install scripts * refactor(dummy_problem): handle ListConfig * update dependencies * refactor(Makefile): more commands * refactor: rename budget -> fidelity * refactor: rename problem -> task or objective function * fix(smacconfig) * format(notebooks): ruff * format * refactor: scenario -> task_type * refactor(hebo): deactivate due to install issues * fix(hpob): path to hpob surrogate files * ignore more * fix(hpobench_config) * fix(pymoo_config) * refactor(yahpo_tasks): regenerate * fix(hpobench): installation (container building) * fix(hpobench): configs, build container * refactor(hpobench): configs * refactor(subselection): update config files * refactor(hpobench_configs): tab -> tabular * refactor(subselection_blackbox): update to new tasks * build: update requirements * Update carps version * Update CHANGELOG.md * add todo * Refactor/installation (#176) * build/refactor(installation): use make commands from anywhere * Update README.md * refactor(install): benchmarks with data * refactor: validate ranking * refactor(yahpo repo) * refactor * refactor(env_vars): define carps root * refactor(Hpobenchconfigs): fix objectives * Reactvate SMAC check * build(Makefile): install swig for smac via pip * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * refactor(subset_configs) * build(nevergrad): upgrade cma to use np2.0 * style(hpobench): add arg to docstring * fix(conditional_search_space): deactivate wrongly active HPs before passing to objective function evlauate * fix(check_missing): status check * refactor(database): more ids * fix(create_cluster_configs): update task info * docs(installation): update * ignore more * refactor/feat: run from database * refactor: move experimenter files to own folder * refactor: move database utils * refactor(: run from db * refactor: container folder -> experimenter folder * feat(experimenter): show stats and export error configs * refactor(assets): shorten filename * feat: Scrape results to database (#178) * fix(gather_data): dict accesses, lists, task_id * fix(create_cluster_configs): dict accesses * feat(scrape_to_db): Scrape file-logged results to database * Update README.md * Sleeker type handling and happy pre-commit :) * Refactor * Quickdev (#179) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#180) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#181) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * Fix/make (#185) * fix(Makefile): access works Makefile did not get copied into * move container_recipes * style(prepare_nas_benchmarks): add docstr * refactor: rename container recipe paths * fix: install paths * fix(install_yahpo): pip install * build(pyproject.toml): include recipe data * Update README.md * Update CHANGELOG.md * Update docs (#186) * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * Quickdev (#193) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit --------- Co-authored-by: Copilot <[email protected]> * Beautify CD Diagram (#194) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() --------- Co-authored-by: Copilot <[email protected]> * Fix/plotting (#196) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() * fix(generate_report): cd plot: names got accidentally sorted, rank results wants more kwargs * Update carps/analysis/utils.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * Update CHANGELOG.md * ignore more * Increase version number * Quickdev (#198) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() * refactor(mo) --------- Co-authored-by: Copilot <[email protected]> * Fix yahpo error (#201) * fix(yahpo): onnx runtime issues * fix(yahpo): issue was local config * Update CHANGELOG.md * Update carps/objective_functions/yahpo.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * Fix missing requirement files (#202) * fix(requirements): readd files * remove pip freeze files * Update CHANGELOG.md * Update version number (#204) --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> Co-authored-by: thibautklenke <[email protected]> Co-authored-by: Lukas Fehring <[email protected]> Co-authored-by: Copilot <[email protected]>
v1.0.3 (#197) * Fix HPOBench (#158) * Fix tabular benchmark not working * Update hpobench install * Add script to fix nas container error * Update CHANGELOG.md * Optuna: add new MO config (#160) * Rename config * Rename config * Add Optuna config * Simplify init of optuna study * Update * Small fix * Update optimizer configs (#161) * feat(skopt): more variants * Add more kwargs for DEHB * Add paper source * Add task info for dummy problem * Update HEBO config * Rework Skopt config * Use base config for synetune * Add comment * restrict ConfigSpace for now (yahpo and smac struggle) * Clean SMAC configs * Remove smac1.4 configs (clash with ConfigSpace, would only work with containers) * Update optuna configs * Update Nevergrad configs / clean * Update CHANGELOG.md * More Pymoo Problems (#162) * Update * Generate pymoo problems * Update pymoo problems * Update CHANGELOG.md * Update HEBO & BBOB-Vizier (#163) * feat(benchmark): add bbob 20d 100 vizier * fix(hebo): sobol scrambling seed default * refactor(hebo): use global seed for sobol * refactor(hebo): update config to pass global seed to scramble * feat(notebook): add comparison of SMAC, HEBO, RS on BBOB20d * Update CHANGELOG.md * Merge * feat(Ax): Add Ax optimizer (SO + MO) (#166) * feat(Ax): Add Ax optimizer (SO + MO) * Update CHANGELOG.md * feat(Ax): Type refinements, fixed OrdinalHyperparameter, added note regarding random seed --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> * fix: SMAC callbacks (#167) * Update CHANGELOG.md * fix(ax): conversion of ordinal HPs * fix(ax): cat HP * fix(ax): allow_inactive_with_values=True * fix(smac20): remove unrecognized arg from TrialValue * fix(ax): typos * Research/subset (#164) * Ignore more * Update * Add notebook * Update * feat: validate ranking * Update * add notebook * Update * Do not plot * Update * Update notebook * Format * Compare subselections * Also save run cfgs * Update notebook * fix timestamp * goodbye bash scripts * make prettier * goosbye old file * goodbye * build(pyproject): update ruff setting * Ignore more * update program * feat(subselect): finally as python * tiny fixes * refactor(subselection) * fix(problem): generate pymoo config * local parallel * refactor(create_subset_config): del dir, better error msg, new cmd * feat: new subsets for BB + MO * run more pymoo problems * Rename problem * refactor(generate_problems): MO * refactor(benchmark_footprint): notebook * refactor(inspect_problems): update notebook * build(yahpo_install): update * update yahpo install * update notebooks * revert subselection progress moved to branch feat/subselection * please pandas * please pandas * fix(hebo): ordinal hyperparameter + precommit * refactor(yahpo): update ConfigSpace API * ignore more * add scenario and subset_id to config * add scenario and subset id to subselection configs * remove duplicates in config * refactor(gather_data): more config keys * feat(gather_data): calculate log performance * update notebook * feat(report) * fix(autorank): api * ignore more * feat(color_palette): more and nicer colors * feat(generate_report): silent plotting * fix(ax): conversion of ordinal HPs * style(utils): pre-commit * style(...): pre-commit * style * fix(ax): cat HP * refactor(generate_report) * fix(ax): allow_inactive_with_values=True * fix(yahpo): ConfigSpace deprecation warning * docs(generate_report): more info * fix(run_autorank): if nothing is lost * fix(utils): filter by final performance: if there is a small max val correctly limit * feat(gather_data): collect from several folders * refactor(generate_report): goodbye plots, fix ranks and norm * style(file_logger) * style(pareto_front) * style(overriderfinde) * style(loggingutils) * style(index_configs) * refactor(generate_report): fun arg default * fix(run_autorank): log msg pos * pre-commit 1 * refactor/fix(nevergrad): pass trial seed to trial info * Remove DummyOptimizer because Random Search behaves the same * Remove DummyOptimizer because Random Search behaves the same * style: make ruff happy * refactor(task): set n_objectives as 1 per default * refactor(synetune): set max budget (fidelity) as fixed hp in synetune * Please mypy nr 1 * refactor(install_yahpo): control root of carps * fix(task): dataclass unmutable arg * fix(task): dataclass unmutable arg * bbob: update ioh requirements * style: please mypy * Update CHANGELOG.md * 170 update nomenclature (#175) * refactor(objective_functions): rename folder, rename file * refactor: Problem -> ObjectiveFunction * refactor: import objective_function * refactor: problem_id -> task_id * refactor: carps.benchmarks -> carps.objective_functions * refactor(task) * refactor(installation) * refactor(install): update * current * refactor(BREAKING): new task definition * refactor: make task instantiable * refactor: generate tasks, new configs * refactor(optimizer): add metadata about mo/mf * tests(optimizers): add * testing * fix install scripts * refactor(dummy_problem): handle ListConfig * update dependencies * refactor(Makefile): more commands * refactor: rename budget -> fidelity * refactor: rename problem -> task or objective function * fix(smacconfig) * format(notebooks): ruff * format * refactor: scenario -> task_type * refactor(hebo): deactivate due to install issues * fix(hpob): path to hpob surrogate files * ignore more * fix(hpobench_config) * fix(pymoo_config) * refactor(yahpo_tasks): regenerate * fix(hpobench): installation (container building) * fix(hpobench): configs, build container * refactor(hpobench): configs * refactor(subselection): update config files * refactor(hpobench_configs): tab -> tabular * refactor(subselection_blackbox): update to new tasks * build: update requirements * Update carps version * Update CHANGELOG.md * add todo * Refactor/installation (#176) * build/refactor(installation): use make commands from anywhere * Update README.md * refactor(install): benchmarks with data * refactor: validate ranking * refactor(yahpo repo) * refactor * refactor(env_vars): define carps root * refactor(Hpobenchconfigs): fix objectives * Reactvate SMAC check * build(Makefile): install swig for smac via pip * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * refactor(subset_configs) * build(nevergrad): upgrade cma to use np2.0 * style(hpobench): add arg to docstring * fix(conditional_search_space): deactivate wrongly active HPs before passing to objective function evlauate * fix(check_missing): status check * refactor(database): more ids * fix(create_cluster_configs): update task info * docs(installation): update * ignore more * refactor/feat: run from database * refactor: move experimenter files to own folder * refactor: move database utils * refactor(: run from db * refactor: container folder -> experimenter folder * feat(experimenter): show stats and export error configs * refactor(assets): shorten filename * feat: Scrape results to database (#178) * fix(gather_data): dict accesses, lists, task_id * fix(create_cluster_configs): dict accesses * feat(scrape_to_db): Scrape file-logged results to database * Update README.md * Sleeker type handling and happy pre-commit :) * Refactor * Quickdev (#179) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#180) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#181) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * Fix/make (#185) * fix(Makefile): access works Makefile did not get copied into * move container_recipes * style(prepare_nas_benchmarks): add docstr * refactor: rename container recipe paths * fix: install paths * fix(install_yahpo): pip install * build(pyproject.toml): include recipe data * Update README.md * Update CHANGELOG.md * Update docs (#186) * Quickdev (#193) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit --------- Co-authored-by: Copilot <[email protected]> * Beautify CD Diagram (#194) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() --------- Co-authored-by: Copilot <[email protected]> * Fix/plotting (#196) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() * fix(generate_report): cd plot: names got accidentally sorted, rank results wants more kwargs * Update carps/analysis/utils.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * Update CHANGELOG.md * ignore more * Increase version number * rm file --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> Co-authored-by: thibautklenke <[email protected]> Co-authored-by: Lukas Fehring <[email protected]> Co-authored-by: Copilot <[email protected]>
v1.0.2 (#195) * Fix HPOBench (#158) * Fix tabular benchmark not working * Update hpobench install * Add script to fix nas container error * Update CHANGELOG.md * Optuna: add new MO config (#160) * Rename config * Rename config * Add Optuna config * Simplify init of optuna study * Update * Small fix * Update optimizer configs (#161) * feat(skopt): more variants * Add more kwargs for DEHB * Add paper source * Add task info for dummy problem * Update HEBO config * Rework Skopt config * Use base config for synetune * Add comment * restrict ConfigSpace for now (yahpo and smac struggle) * Clean SMAC configs * Remove smac1.4 configs (clash with ConfigSpace, would only work with containers) * Update optuna configs * Update Nevergrad configs / clean * Update CHANGELOG.md * More Pymoo Problems (#162) * Update * Generate pymoo problems * Update pymoo problems * Update CHANGELOG.md * Update HEBO & BBOB-Vizier (#163) * feat(benchmark): add bbob 20d 100 vizier * fix(hebo): sobol scrambling seed default * refactor(hebo): use global seed for sobol * refactor(hebo): update config to pass global seed to scramble * feat(notebook): add comparison of SMAC, HEBO, RS on BBOB20d * Update CHANGELOG.md * Merge * feat(Ax): Add Ax optimizer (SO + MO) (#166) * feat(Ax): Add Ax optimizer (SO + MO) * Update CHANGELOG.md * feat(Ax): Type refinements, fixed OrdinalHyperparameter, added note regarding random seed --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> * fix: SMAC callbacks (#167) * Update CHANGELOG.md * fix(ax): conversion of ordinal HPs * fix(ax): cat HP * fix(ax): allow_inactive_with_values=True * fix(smac20): remove unrecognized arg from TrialValue * fix(ax): typos * Research/subset (#164) * Ignore more * Update * Add notebook * Update * feat: validate ranking * Update * add notebook * Update * Do not plot * Update * Update notebook * Format * Compare subselections * Also save run cfgs * Update notebook * fix timestamp * goodbye bash scripts * make prettier * goosbye old file * goodbye * build(pyproject): update ruff setting * Ignore more * update program * feat(subselect): finally as python * tiny fixes * refactor(subselection) * fix(problem): generate pymoo config * local parallel * refactor(create_subset_config): del dir, better error msg, new cmd * feat: new subsets for BB + MO * run more pymoo problems * Rename problem * refactor(generate_problems): MO * refactor(benchmark_footprint): notebook * refactor(inspect_problems): update notebook * build(yahpo_install): update * update yahpo install * update notebooks * revert subselection progress moved to branch feat/subselection * please pandas * please pandas * fix(hebo): ordinal hyperparameter + precommit * refactor(yahpo): update ConfigSpace API * ignore more * add scenario and subset_id to config * add scenario and subset id to subselection configs * remove duplicates in config * refactor(gather_data): more config keys * feat(gather_data): calculate log performance * update notebook * feat(report) * fix(autorank): api * ignore more * feat(color_palette): more and nicer colors * feat(generate_report): silent plotting * fix(ax): conversion of ordinal HPs * style(utils): pre-commit * style(...): pre-commit * style * fix(ax): cat HP * refactor(generate_report) * fix(ax): allow_inactive_with_values=True * fix(yahpo): ConfigSpace deprecation warning * docs(generate_report): more info * fix(run_autorank): if nothing is lost * fix(utils): filter by final performance: if there is a small max val correctly limit * feat(gather_data): collect from several folders * refactor(generate_report): goodbye plots, fix ranks and norm * style(file_logger) * style(pareto_front) * style(overriderfinde) * style(loggingutils) * style(index_configs) * refactor(generate_report): fun arg default * fix(run_autorank): log msg pos * pre-commit 1 * refactor/fix(nevergrad): pass trial seed to trial info * Remove DummyOptimizer because Random Search behaves the same * Remove DummyOptimizer because Random Search behaves the same * style: make ruff happy * refactor(task): set n_objectives as 1 per default * refactor(synetune): set max budget (fidelity) as fixed hp in synetune * Please mypy nr 1 * refactor(install_yahpo): control root of carps * fix(task): dataclass unmutable arg * fix(task): dataclass unmutable arg * bbob: update ioh requirements * style: please mypy * Update CHANGELOG.md * 170 update nomenclature (#175) * refactor(objective_functions): rename folder, rename file * refactor: Problem -> ObjectiveFunction * refactor: import objective_function * refactor: problem_id -> task_id * refactor: carps.benchmarks -> carps.objective_functions * refactor(task) * refactor(installation) * refactor(install): update * current * refactor(BREAKING): new task definition * refactor: make task instantiable * refactor: generate tasks, new configs * refactor(optimizer): add metadata about mo/mf * tests(optimizers): add * testing * fix install scripts * refactor(dummy_problem): handle ListConfig * update dependencies * refactor(Makefile): more commands * refactor: rename budget -> fidelity * refactor: rename problem -> task or objective function * fix(smacconfig) * format(notebooks): ruff * format * refactor: scenario -> task_type * refactor(hebo): deactivate due to install issues * fix(hpob): path to hpob surrogate files * ignore more * fix(hpobench_config) * fix(pymoo_config) * refactor(yahpo_tasks): regenerate * fix(hpobench): installation (container building) * fix(hpobench): configs, build container * refactor(hpobench): configs * refactor(subselection): update config files * refactor(hpobench_configs): tab -> tabular * refactor(subselection_blackbox): update to new tasks * build: update requirements * Update carps version * Update CHANGELOG.md * add todo * Refactor/installation (#176) * build/refactor(installation): use make commands from anywhere * Update README.md * refactor(install): benchmarks with data * refactor: validate ranking * refactor(yahpo repo) * refactor * refactor(env_vars): define carps root * refactor(Hpobenchconfigs): fix objectives * Reactvate SMAC check * build(Makefile): install swig for smac via pip * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * refactor(subset_configs) * build(nevergrad): upgrade cma to use np2.0 * style(hpobench): add arg to docstring * fix(conditional_search_space): deactivate wrongly active HPs before passing to objective function evlauate * fix(check_missing): status check * refactor(database): more ids * fix(create_cluster_configs): update task info * docs(installation): update * ignore more * refactor/feat: run from database * refactor: move experimenter files to own folder * refactor: move database utils * refactor(: run from db * refactor: container folder -> experimenter folder * feat(experimenter): show stats and export error configs * refactor(assets): shorten filename * feat: Scrape results to database (#178) * fix(gather_data): dict accesses, lists, task_id * fix(create_cluster_configs): dict accesses * feat(scrape_to_db): Scrape file-logged results to database * Update README.md * Sleeker type handling and happy pre-commit :) * Refactor * Quickdev (#179) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#180) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#181) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * Fix/make (#185) * fix(Makefile): access works Makefile did not get copied into * move container_recipes * style(prepare_nas_benchmarks): add docstr * refactor: rename container recipe paths * fix: install paths * fix(install_yahpo): pip install * build(pyproject.toml): include recipe data * Update README.md * Update CHANGELOG.md * Update docs (#186) * Quickdev (#193) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit --------- Co-authored-by: Copilot <[email protected]> * Beautify CD Diagram (#194) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * fix(process_logs): process all rows not just incumbents only incumbents is not possible because at this point we do not now the hypervolume and thus cannot compute the incumbent with confidence * fix(database): column names/experiment ids before: n_trials was an experiment id but is later used to log the evaluated number of trials and thus the information is lost * fix(run_autorank): pass required arg * refactor(utils): move function to avoid circular import * refactor(calc_hypervolume): separate into functions, add functions * fix(gather_data): MO processing hypervolume is only calculated in `normalize_logs`! * fix(gather_dat): keep task.optimization.n_trials as task column * fix(gather_data): path to save logs * fix(gather_data): only determine task type if info not present * refactor(gather_data): specify outdir * refactor(gather_data): specify outdir * fix(install): mfpbench missing requirements * fix(synetune): accidently increased trial_counter -> ST only had half of the budget * fix(run_autorank): sample_matrix arg * fix(gather_data): cost conversion * fix(analysis): calculation of final performance * refactor(utils): add analysis helper functions * refactor(check_missing): handle yaml errors, specify n_processes * refactor(calc_runtime): adapt to new structure * fix(gather_data): recalculate mo ids bc they change after calculating HV * refactor(calc_hypervolum): use solution writing groups to filesystem to avoid memory issues * refactor(generate_report): replot * refactor(generate_report): small * fix(gather_data): check of mo runs existence * refactor(nevergrad): unique trial name as a function * refactor(trials): pass additional info in trial_info * refactor(py_experimenter): update config * feat(logging): log python env info to file * style(carps/analysis/utils.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(calc_hypervolume): remove faulty entry point * style(carps/analysis/calc_hypervolume.py): fix typo Co-authored-by: Copilot <[email protected]> * refactor(file_logger): add safety guard if directory is None * refactor(carps/analysis/run_autorank.py) Inconsistent indexing: sorted_ranks[r] may not align with positions. Use sorted_ranks.iloc[r] for both elements to ensure correct lookups. Co-authored-by: Copilot <[email protected]> * style(run_autorank): fix pre-commit * refactor(plotting): beautify CD diagram * refactor(run_autorank): remove spurious plt.show() --------- Co-authored-by: Copilot <[email protected]> * update version v1.0.1 -> v1.0.2 * Update MANIFEST.in * refactor(loggingutils): show warning if pip is not found and add TODO to also discover uv later --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> Co-authored-by: thibautklenke <[email protected]> Co-authored-by: Lukas Fehring <[email protected]> Co-authored-by: Copilot <[email protected]>
v1.0.1 (#187) * Fix HPOBench (#158) * Fix tabular benchmark not working * Update hpobench install * Add script to fix nas container error * Update CHANGELOG.md * Optuna: add new MO config (#160) * Rename config * Rename config * Add Optuna config * Simplify init of optuna study * Update * Small fix * Update optimizer configs (#161) * feat(skopt): more variants * Add more kwargs for DEHB * Add paper source * Add task info for dummy problem * Update HEBO config * Rework Skopt config * Use base config for synetune * Add comment * restrict ConfigSpace for now (yahpo and smac struggle) * Clean SMAC configs * Remove smac1.4 configs (clash with ConfigSpace, would only work with containers) * Update optuna configs * Update Nevergrad configs / clean * Update CHANGELOG.md * More Pymoo Problems (#162) * Update * Generate pymoo problems * Update pymoo problems * Update CHANGELOG.md * Update HEBO & BBOB-Vizier (#163) * feat(benchmark): add bbob 20d 100 vizier * fix(hebo): sobol scrambling seed default * refactor(hebo): use global seed for sobol * refactor(hebo): update config to pass global seed to scramble * feat(notebook): add comparison of SMAC, HEBO, RS on BBOB20d * Update CHANGELOG.md * Merge * feat(Ax): Add Ax optimizer (SO + MO) (#166) * feat(Ax): Add Ax optimizer (SO + MO) * Update CHANGELOG.md * feat(Ax): Type refinements, fixed OrdinalHyperparameter, added note regarding random seed --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> * fix: SMAC callbacks (#167) * Update CHANGELOG.md * fix(ax): conversion of ordinal HPs * fix(ax): cat HP * fix(ax): allow_inactive_with_values=True * fix(smac20): remove unrecognized arg from TrialValue * fix(ax): typos * Research/subset (#164) * Ignore more * Update * Add notebook * Update * feat: validate ranking * Update * add notebook * Update * Do not plot * Update * Update notebook * Format * Compare subselections * Also save run cfgs * Update notebook * fix timestamp * goodbye bash scripts * make prettier * goosbye old file * goodbye * build(pyproject): update ruff setting * Ignore more * update program * feat(subselect): finally as python * tiny fixes * refactor(subselection) * fix(problem): generate pymoo config * local parallel * refactor(create_subset_config): del dir, better error msg, new cmd * feat: new subsets for BB + MO * run more pymoo problems * Rename problem * refactor(generate_problems): MO * refactor(benchmark_footprint): notebook * refactor(inspect_problems): update notebook * build(yahpo_install): update * update yahpo install * update notebooks * revert subselection progress moved to branch feat/subselection * please pandas * please pandas * fix(hebo): ordinal hyperparameter + precommit * refactor(yahpo): update ConfigSpace API * ignore more * add scenario and subset_id to config * add scenario and subset id to subselection configs * remove duplicates in config * refactor(gather_data): more config keys * feat(gather_data): calculate log performance * update notebook * feat(report) * fix(autorank): api * ignore more * feat(color_palette): more and nicer colors * feat(generate_report): silent plotting * fix(ax): conversion of ordinal HPs * style(utils): pre-commit * style(...): pre-commit * style * fix(ax): cat HP * refactor(generate_report) * fix(ax): allow_inactive_with_values=True * fix(yahpo): ConfigSpace deprecation warning * docs(generate_report): more info * fix(run_autorank): if nothing is lost * fix(utils): filter by final performance: if there is a small max val correctly limit * feat(gather_data): collect from several folders * refactor(generate_report): goodbye plots, fix ranks and norm * style(file_logger) * style(pareto_front) * style(overriderfinde) * style(loggingutils) * style(index_configs) * refactor(generate_report): fun arg default * fix(run_autorank): log msg pos * pre-commit 1 * refactor/fix(nevergrad): pass trial seed to trial info * Remove DummyOptimizer because Random Search behaves the same * Remove DummyOptimizer because Random Search behaves the same * style: make ruff happy * refactor(task): set n_objectives as 1 per default * refactor(synetune): set max budget (fidelity) as fixed hp in synetune * Please mypy nr 1 * refactor(install_yahpo): control root of carps * fix(task): dataclass unmutable arg * fix(task): dataclass unmutable arg * bbob: update ioh requirements * style: please mypy * Update CHANGELOG.md * 170 update nomenclature (#175) * refactor(objective_functions): rename folder, rename file * refactor: Problem -> ObjectiveFunction * refactor: import objective_function * refactor: problem_id -> task_id * refactor: carps.benchmarks -> carps.objective_functions * refactor(task) * refactor(installation) * refactor(install): update * current * refactor(BREAKING): new task definition * refactor: make task instantiable * refactor: generate tasks, new configs * refactor(optimizer): add metadata about mo/mf * tests(optimizers): add * testing * fix install scripts * refactor(dummy_problem): handle ListConfig * update dependencies * refactor(Makefile): more commands * refactor: rename budget -> fidelity * refactor: rename problem -> task or objective function * fix(smacconfig) * format(notebooks): ruff * format * refactor: scenario -> task_type * refactor(hebo): deactivate due to install issues * fix(hpob): path to hpob surrogate files * ignore more * fix(hpobench_config) * fix(pymoo_config) * refactor(yahpo_tasks): regenerate * fix(hpobench): installation (container building) * fix(hpobench): configs, build container * refactor(hpobench): configs * refactor(subselection): update config files * refactor(hpobench_configs): tab -> tabular * refactor(subselection_blackbox): update to new tasks * build: update requirements * Update carps version * Update CHANGELOG.md * add todo * Refactor/installation (#176) * build/refactor(installation): use make commands from anywhere * Update README.md * refactor(install): benchmarks with data * refactor: validate ranking * refactor(yahpo repo) * refactor * refactor(env_vars): define carps root * refactor(Hpobenchconfigs): fix objectives * Reactvate SMAC check * build(Makefile): install swig for smac via pip * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * fix(yahpo_configs): convert config space from hydra/omegaconf to native python objects * refactor(subset_configs) * build(nevergrad): upgrade cma to use np2.0 * style(hpobench): add arg to docstring * fix(conditional_search_space): deactivate wrongly active HPs before passing to objective function evlauate * fix(check_missing): status check * refactor(database): more ids * fix(create_cluster_configs): update task info * docs(installation): update * ignore more * refactor/feat: run from database * refactor: move experimenter files to own folder * refactor: move database utils * refactor(: run from db * refactor: container folder -> experimenter folder * feat(experimenter): show stats and export error configs * refactor(assets): shorten filename * feat: Scrape results to database (#178) * fix(gather_data): dict accesses, lists, task_id * fix(create_cluster_configs): dict accesses * feat(scrape_to_db): Scrape file-logged results to database * Update README.md * Sleeker type handling and happy pre-commit :) * Refactor * Quickdev (#179) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#180) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * Quickdev (#181) * refactor: subselection notebooks * refactor(run_autorank): directly pass df crit * build(pyproject): remove all extras * update notebook * fix(legacy_task_ids) * feat: create database experiments via slurm * refactor/feat: update subselection, add python methods * format(shift_v2nobrute.c): finally * feat(subselect): make usable * fix(yahpo): generation of configs, set time budget to none * refactor(create_cluster_configs): calculate config hash differently * feat(create_cluster_configs_subselection) * fix(file_logger): logging msg * fix(yahpo_task_gen): output space * fix(yahpo output space) * feat(show_stats): split errors into yahpo and non yahpo, save error message * fix/build: hpobench benchmark install * refactor(database_logger): also log n function calls for incumbents * refactor(database_logger): also log n function calls for incumbents * feat: download and process results from database * feat(show_stats): split error msgs in known and unknown * fix(nevergrad): optimizer config (one param needs to be float and somehow the conversion from yaml to json str back to DictConfig messes this up) * refactor(Makefile): for using uv, make sure pip is installed and accessible * fix(synetune): passing metrics * fix(nevergrad): config out of bounds (just a tiny bit) * refactor(create_cluster_configs_subselection) * fix(Ax): always convert cost to float * fix(Ax): hopefully numeric mean bounds * feat(reset_experiments): reset only yahpo attr none type errors * ignore more * refactor(show_stats): filter more errors * refactor(create_cluster_configs): separate hydra from functionality to make it importable from other scripts * refactor(create_cluster_configs): rename variables * refactor(gather_data): handle logs from database * docs(process_logs): add runcommand * feat(reset_experiments): add more reset options * fix(reset_experiments): correctly delete falsely done experiments * refactor(generate_report) * refactor(generate_report): small plotting adjustments * fix(run_autorank): class args * fix(yahpo): 1st to fix yahpo nonetype error Copy onnx model. * refactor(check_missing): adapt to new structure * refactor(gather_data): adapt to new structure * refactor(pyproject.toml): add pyexperimenter requirement * refactor(scrape_results_to_database) * refactor(Makefile): upgrade numpy * feat(container): add recipe for virtual env * refactor(yahpo install) * refactor(requirements) * refactor(README): adjust paths * refactor(download_results): add more infos * docs(README): add info about container * fix(unique task ids) * fix(dummy_problem): passing of configuration space * fix(synetune): passing metric for SyncMOBSTER * fix/build(mfpbench): new xgboost for pd1 * fix(test_tasks): query fidelity * fix(pyproject): ONLY COPY CARPS FILES 😭👽 * fix(pyproject.toml) email adress * fix/build: properly include files in package (not too many) * update README.md * docs(reset_experiments): add convenience cmds in docstring * refactor(show_stats): print number of known errors * Fix/make (#185) * fix(Makefile): access works Makefile did not get copied into * move container_recipes * style(prepare_nas_benchmarks): add docstr * refactor: rename container recipe paths * fix: install paths * fix(install_yahpo): pip install * build(pyproject.toml): include recipe data * Update README.md * Update CHANGELOG.md * Update docs (#186) * Update CHANGELOG.md * Update carps version 1.0.0 -> 1.0.1 --------- Co-authored-by: thibautklenke <[email protected]> Co-authored-by: benjamc <[email protected]> Co-authored-by: thibautklenke <[email protected]> Co-authored-by: Lukas Fehring <[email protected]>
v0.1.1 (#151) * fix: typo * build: update scripts * ignore python * docs: update install instructions * docs: update install instructions * build: fix makefile typo * docs: fix typos * build: updates for hpobench installation * build: updates for hpobench installation * Add logo to README.md + docs; adapt docs theme colors to logo * Actually add the logo file * fix: changelog entry for initial version * add running data to readme (#139) * add running data to readme * Update README.md --------- Co-authored-by: benjamc <[email protected]> * Update plotting * Update filepath * fix(check_missing.py): Fix criterion for max trials (#150) For MF, max trials can be a mu above integer value specified because of the partial trials. Fix the condition. * Fix HEBO integration (#149) * fix(hebo): Do not increase trial_counter twice Must have been forgotton 😭💀 * Update CHANGELOG.md * Update CHANGELOG.md * Update version number 0.1.0->0.1.1 --------- Co-authored-by: Sarah Krebs <[email protected]> Co-authored-by: Helena Graf <[email protected]> Co-authored-by: Sarah Segel <[email protected]> Co-authored-by: Difan Deng <[email protected]>