fix(pipeline): Ensure optimizer is updated with report #261
Conversation
| if isinstance(optimizer, Optimizer): | ||
| _optimizer = optimizer | ||
| else: |
There was a problem hiding this comment.
The main changes here is that previously we would always construct an optimizer, I just shifted the previous code into the else block while using the if part to check if we have an already instantiated one.
| optimizer: ( | ||
| type[Optimizer] | Optimizer.CreateSignature | Optimizer | None | ||
| ) = None, |
There was a problem hiding this comment.
Now accept already instantiated optimizer (Optimizer)
|
|
||
| @task.on_result | ||
| def tell_optimizer(_: Any, report: Trial.Report) -> None: | ||
| _optimizer.tell(report) | ||
|
|
There was a problem hiding this comment.
Whoops, this should have been here...
| class MyOptimizer(SMACOptimizer): | ||
| def __init__( | ||
| self, | ||
| *args: Any, | ||
| **kwargs: Any, | ||
| ) -> None: | ||
| self.told_report: Trial.Report | None = None | ||
| super().__init__(*args, **kwargs) | ||
|
|
||
| @override | ||
| def tell(self, report: Trial.Report) -> None: | ||
| self.told_report = report | ||
| return super().tell(report) |
There was a problem hiding this comment.
Custom optimizer we can inspect post fact, it just records the report it was tell'ed about, defaulting to SMAC optimizer for everything else.
| component = Component(object, space={"a": (0.0, 1.0)}) | ||
| optimizer = MyOptimizer.create( | ||
| space=component, | ||
| metrics=METRIC, | ||
| bucket=PathBucket(tmp_path), | ||
| ) | ||
|
|
||
| history = component.optimize( | ||
| target_funtion, | ||
| metric=METRIC, | ||
| optimizer=optimizer, | ||
| max_trials=1, | ||
| working_dir=tmp_path, | ||
| ) | ||
|
|
||
| assert optimizer.told_report is history[0] |
There was a problem hiding this comment.
Pass in the instantiated optimizer and ensure that the report it was told about is exactly the same object that is recorded in the history.
Fixes a bug where using
pipeline.optimizewas not actually calling thetell()of the optimizer. Added a test to ensure this does not get removed.@LennartPurucker tagged for review, no need to if you don't have time as I unfortunately need to merge this now. I've confirmed it works with the tests but also in the other setting via good ol' print debugging. Changes are annotated to expedite the process.