Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@eddiebergman
Copy link
Contributor

If a task which was used for a Trial, such as those when optimizing trials from an Optimizer, exceeded it's constraints, it would previously just be a generic Task exception which would have to be handled with @task.on_exception. However it's often the case that we would rather get back a Trial.fail() through @task.on_result such that we can handle it in a uniform manner with other trials that failed. This also caused issues with the scheduler error handling mechanism for which #213 was introduced.

The new default behaviour of the PynisherPlugin is to auto-detect if a Task contains a Trial. If so and the Task exceeded one of it's constraints, it would simply return task.fail() with the relevant pynisher exception filled in. This behavior can be disabled with disable_trial_handling.


Take for example this function, we would like to report that the trial failed if it timed out during post-processing or caused a memory error during post processing, but we would also like to have a ValueError from human error still be classed as a @task.on_exception

def fn(trial: Trial) -> Trial.Report:
	with trial.begin():
		# all good
		pass

	if trial.exception:
		return trial.fail()

	# !!! Memory exception or time exception raised here !!!
	# We likely want this to go under trial.fail()
	do_some_post_processing()

	# Value error from human error which we want to have
	# raised properly and not return a trial.fail() report
	1/0

	return trial.success()
	
reports = []
task = Task(fn, plugins=PynisherPlugin(...))

To handle errors as described above, you would do:

@scheduler.on_start
def submit_trials():
    trial = get_next_trial()
    task.submit(trial)

@task.on_result
def handle_result(future, report: Trial.Report):
    match report.status:
    	case Trial.SUCCESS:
			print("trial success")
		case Trial.FAIL:
			print("trial failed")
	
    reports.add(report)
    
@task.on_exception
def handle_exception(_, exception: Exception):
	print("An unexpected exception occured!")

The previous work-around may have looked like this:

reports = []
pending_trials: dict[Future, Trial] = {}

@scheduler.on_start
def submit_trials():
    trial = get_next_trial()
    future = task.submit(trial)
	self.pending_trials[future] = trial    

@task.on_result
def handle_result(future, report: Trial.Report):
	pending_trials.pop(report, None)
    match report.status:
    	case Trial.SUCCESS:
			print("trial success")
		case Trial.FAIL:
			print("trial failed")
	
    reports.add(report)
    
@task.on_exception
def handle_exception(future: Future, exception: Exception):
	trial = pending_trials.get(future)
   	if trial is not None and isinstance(trial.exception, PynisherPlugin.PynisherException):
			trial.exception = exception
	  		# How to access traceback here??
	       	fail_report = trial.fail()
			print("trial failed outside begin() block")
			return handle_result(future, fail_report)
	
	# Proceed with handling the syntax error handling you like
    print("An unexpected exception occured!")

@eddiebergman eddiebergman added the feature A new feature label Jan 22, 2024
@eddiebergman eddiebergman self-assigned this Jan 22, 2024
If a task which was used for a `Trial`, such as those when
optimizing trials from an Optimizer, exceeded it's constraints,
it would previously just be a generic `Task` exception. However
it's often the case that we would rather get back a `Trial.fail()`
such that we can handle it in a uniform manner with other trials
that failed.
@eddiebergman eddiebergman force-pushed the feat-pynisher-auto-detect-trial branch from 7400ed9 to 7422982 Compare January 22, 2024 09:52
@eddiebergman eddiebergman merged commit ec6f235 into main Jan 22, 2024
@eddiebergman eddiebergman deleted the feat-pynisher-auto-detect-trial branch January 22, 2024 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants