feat(Pynisher): Detect tasks with Trial to report FAIL
#220
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a task which was used for a
Trial, such as those when optimizing trials from an Optimizer, exceeded it's constraints, it would previously just be a genericTaskexception which would have to be handled with@task.on_exception. However it's often the case that we would rather get back aTrial.fail()through@task.on_resultsuch that we can handle it in a uniform manner with other trials that failed. This also caused issues with the scheduler error handling mechanism for which #213 was introduced.The new default behaviour of the
PynisherPluginis to auto-detect if aTaskcontains aTrial. If so and theTaskexceeded one of it's constraints, it would simply returntask.fail()with the relevant pynisher exception filled in. This behavior can be disabled withdisable_trial_handling.Take for example this function, we would like to report that the trial failed if it timed out during post-processing or caused a memory error during post processing, but we would also like to have a
ValueErrorfrom human error still be classed as a@task.on_exceptionTo handle errors as described above, you would do:
The previous work-around may have looked like this: